Juillet 2014 - Lisa_teatalk - mailman.iro.umontreal.ca

Tea Talk 1 Aug Friday @13.00 AA3195 by Yoshua Bengio
by Kyung Hyun Cho 02 Aoû '14

02 Aoû '14

Dear all, We will have a tea talk this Friday by Prof. Yoshua Bengio. See below for the details and the attached paper. Hope to see many of you there! - Cho === - Speaker: Prof. Yoshua Bengio (University of Montreal) - Date and Time: 1 Aug 2014 @13.00 - Place: AA3195 - Title: How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation - Abstract: In this paper we propose to exploit reconstruction as a layer-local training signal for deep learning, be it generative or discriminant, single or multi-modal, supervised, semi-supervised or unsupervised, feedforward or recurrent. Reconstructions can be propagated in a form of target propagation playing a role similar to back-propagation but helping to reduce the reliance on back-propagation in order to perform credit assignment across many levels of possibly strong non-linearities (which is difficult for back-propagation). A regularized auto-encoder tends produce a reconstruction that is a more likely version of its input, i.e., a small move in the direction of higher likelihood. By generalizing gradients, target propagation may also allow to train deep networks with discrete hidden units. If the auto-encoder takes both a representation of input and target (or of any side information) in input, then its reconstruction of input representation provides a target towards a representation that is more likely, conditioned on all the side information. A deep auto-encoder decoding path generalizes gradient propagation in a learned way that can thus handle not just infinitesimal changes but larger, discrete changes, hopefully allowing credit assignment through a long chain of non-linear operations. For this to work, each layer must be a good denoising or regularized auto-encoder itself. In addition to each layer being a good auto-encoder, the encoder also learns to please the upper layers by transforming the data into a space where it is easier to model by them, flattening manifolds and disentangling factors. The motivations and theoretical justifications for this approach are laid down in this paper, along with conjectures that will have to be verified either mathematically or experimentally.

5 8

Two tea talks this week: Coline Devin on Wednesday and Yoshua Bengio on Friday
by Kyung Hyun Cho 30 Jul '14

30 Jul '14

Dear all, I have some last minute changes to the schedule of tea talks this week: (1) Colin Devin's Tea Talk on Wednesday at 13.00 (AA3195) (2) Yoshua Bengio's Tea Talk on Friday at 13.00 (AA3195) The first one is the already announced one, and there is no change. The second one has been scheduled this morning, and Yoshua will tell us about his new idea on a new learning criterion for deep neural networks. I'm attaching the abstract by Yoshua at the end of this email. Best, - Cho === Speaker: Prof. Yoshua Bengio (University of Montreal) Title: How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation Abstract: We discuss the usefulness and rationality of optimism in general sequential decision making. Humans are often optimistically biased and optimists are achieving more of their ambitions than more rational people. We provide a mathematical analysis of general optimistic agents and identify how they can be better or worse than strictly rational agents. These agents select the most optimistic among the still plausible hypotheses from a class. Further, we discuss a milder form of optimism in the case of continuously parameterized classes. We refer to this setting as reward-modulated inference, a framework that includes recent models for synaptic weight updates in neuroscience as a special case. At the center is a trade-off between assigning high probability to likable or likely events. A nice consequence is that we can formulate generative and discriminative learning as endpoints of a continuum. Finally, I present some first experiments using autoencoders for the classical MNIST task showing a very simple way of getting ok but not state-of-the art accuracy. The aim is not to optimize narrow tasks maximally but to be able to achieve the law of effect, making choices with a frequency proportional to past rewards in situations deemed similar. In really complex tasks this can be a sufficient objective and one that humans and animals often settle for. Also, the strategy provides suitable exploration as well as robustness to change.

1 1

Tea Talk 30 July Wednesday @13.00 AA3195 by Coline Devine
by Kyung Hyun Cho 29 Jul '14

29 Jul '14

Dear all, This Wednesday at 13.00, Coline will tell us about using a neural network to learn continuous space representations of words. Hope to see many of you on Wednesday! - Cho === - Speaker: Coline Devine (Harvey Mudd College) - Date and Time: 30 July 2014 @13.00 - Place: AA3195 - Title: Learning continuous space representations of words

1 0

Tea Talk 25 July Friday @13.00 AA3195 by Bart van Merrienboer
by Kyung Hyun Cho 24 Jul '14

24 Jul '14

Dear all, This Friday at 13.00, Bart will tell us about the statistical machine translation, especially, focusing on the statistical phrase-based machine translation and how neural networks can be integrated into the existing system. >From this week, we're back at our usual teatalk place (AA3195). Hope to see many of you on Friday! - Cho ==== - Speaker: Bart van Merrienboer (University of Warwick, École Polytechnique, Chalmers Institute of Technology, University of Montreal) - Date and Time: 25 July 2014 @13.00 - Place: AA3195 - Title: Statistical Machine Translation and Neural Networks

1 0

Tea Talk 13 July Friday @13.00 Z-209 by Dustin Webb
by Kyung Hyun Cho 17 Jul '14

17 Jul '14

Hi all, This is a reminder for the upcoming tea talk at 13.00 on Friday. *The tea talk (Z-209)* will be followed by *Nicolas Boulanger's Ph.D. defence at AA-3195* at 14.00. I encourage people to attend both the tea talk and the defence. Best, - Cho On Mon, Jul 14, 2014 at 1:50 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > This Friday at 13.00, Dustin will give a tea talk about building a > deep/multilayer kernel based on the paper <Kernel Methods for Deep > Learning> by Youngmin Cho and Lawrence K. Saul presented earlier at NIPS > 2009. > > Note that the place is not the usual lecture hall in the same building, > but is Z-209 in the neighbouring building (Pavillon Claire-McNicoll). > > Hope to see many of you on Friday! > - Cho > > ==== > > - Speaker: Dustin Webb (University of Utah) > - Date and Time: 13 July 2014 @13.00 > - Place: Z-209 > - Abstract: > > We introduce a new family of positive-deﬁnite kernel functions that mimic > the > computation in large, multilayer neural nets. These kernel functions can > be used > in shallow architectures, such as support vector machines (SVMs), or in > deep > kernel-based architectures that we call multilayer kernel machines (MKMs). > We > evaluate SVMs and MKMs with these kernel functions on problems designed to > illustrate the advantages of deep architectures. On several problems, we > obtain > better results than previous, leading benchmarks from both SVMs with > Gaussian > kernels as well as deep belief nets. > > http://cseweb.ucsd.edu/~saul/papers/nips09_kernel.pdf >

3 2

Tea Talk 13 July Friday @13.00 Z-209 by Dustin Webb
by Kyung Hyun Cho 15 Jul '14

15 Jul '14

Dear all, This Friday at 13.00, Dustin will give a tea talk about building a deep/multilayer kernel based on the paper <Kernel Methods for Deep Learning> by Youngmin Cho and Lawrence K. Saul presented earlier at NIPS 2009. Note that the place is not the usual lecture hall in the same building, but is Z-209 in the neighbouring building (Pavillon Claire-McNicoll). Hope to see many of you on Friday! - Cho ==== - Speaker: Dustin Webb (University of Utah) - Date and Time: 13 July 2014 @13.00 - Place: Z-209 - Abstract: We introduce a new family of positive-deﬁnite kernel functions that mimic the computation in large, multilayer neural nets. These kernel functions can be used in shallow architectures, such as support vector machines (SVMs), or in deep kernel-based architectures that we call multilayer kernel machines (MKMs). We evaluate SVMs and MKMs with these kernel functions on problems designed to illustrate the advantages of deep architectures. On several problems, we obtain better results than previous, leading benchmarks from both SVMs with Gaussian kernels as well as deep belief nets. http://cseweb.ucsd.edu/~saul/papers/nips09_kernel.pdf

3 2

(Date Fixed) Tea Talk 18 July Friday @13.00 Z-209 by Dustin Webb
by Kyung Hyun Cho 15 Jul '14

15 Jul '14

Sorry, everyone! It's not 13 July but "18 July"! On Mon, Jul 14, 2014 at 1:50 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > This Friday at 13.00, Dustin will give a tea talk about building a > deep/multilayer kernel based on the paper <Kernel Methods for Deep > Learning> by Youngmin Cho and Lawrence K. Saul presented earlier at NIPS > 2009. > > Note that the place is not the usual lecture hall in the same building, > but is Z-209 in the neighbouring building (Pavillon Claire-McNicoll). > > Hope to see many of you on Friday! > - Cho > > ==== > > - Speaker: Dustin Webb (University of Utah) > - Date and Time: 13 July 2014 @13.00 > - Place: Z-209 > - Abstract: > > We introduce a new family of positive-deﬁnite kernel functions that mimic > the > computation in large, multilayer neural nets. These kernel functions can > be used > in shallow architectures, such as support vector machines (SVMs), or in > deep > kernel-based architectures that we call multilayer kernel machines (MKMs). > We > evaluate SVMs and MKMs with these kernel functions on problems designed to > illustrate the advantages of deep architectures. On several problems, we > obtain > better results than previous, leading benchmarks from both SVMs with > Gaussian > kernels as well as deep belief nets. > > http://cseweb.ucsd.edu/~saul/papers/nips09_kernel.pdf >

1 0