- Lisa_teatalk - mailman.iro.umontreal.ca

Tea Talk 10 Sep Wednesday @13.30 AA3195 by Matthieu Courbariaux
by Kyung Hyun Cho 08 Sep '14

08 Sep '14

Dear all, Matthieu Courbariaux will tell us about how to program fPGA with OpenCL this Wednesday at 13.30 in a usual tea talk place (AA3195). Best, - Cho

1 0

Thesis Defense by Razvan Pascanu on 3 Sep (Tomorrow!) @12.30 (AA3195)
by Kyung Hyun Cho 03 Sep '14

03 Sep '14

Dear all, This is a reminder about the thesis defense *tomorrow* (3 Sep, Wednesday) by Razvan Pascanu from *12.30*. The defense will take place at *AA3195*. Again, everyone at the lab is highly encouraged to attend! - Cho On Tue, Sep 2, 2014 at 8:21 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > This is a reminder about the thesis defence today by Ian Goodfellow. The > defence will take place at C-3151 (Pavillon Lionel-Groulx) from 13.00. > > Everyone at the lab is encouraged to attend! > - Cho > > > On Mon, Aug 25, 2014 at 10:18 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> > wrote: > >> Dear all, >> >> I'd like announce two thesis defences scheduled next week. >> >> - On Tuesday, 2 Sep 2014 from 13.00 @C-3151, Pavillon Lionel-Groulx >> >> Ian Goodfellow will defend his thesis. The external examiner is Prof. >> Yann LeCun of New York University. >> >> - On Wednesday, 3 Sep 2014 from 12.30 @AA3195 >> >> Razvan Pascanu will defend his thesis. The external examiner is Dr. Ilya >> Sutskever of Google. >> >> I hope to see many of you there! >> - Cho >> >> >

1 0

Thesis Defence by Ian Goodfellow on 2 Sep (Today) @13.00 (C-3151)
by Kyung Hyun Cho 02 Sep '14

02 Sep '14

Dear all, This is a reminder about the thesis defence today by Ian Goodfellow. The defence will take place at C-3151 (Pavillon Lionel-Groulx) from 13.00. Everyone at the lab is encouraged to attend! - Cho On Mon, Aug 25, 2014 at 10:18 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > I'd like announce two thesis defences scheduled next week. > > - On Tuesday, 2 Sep 2014 from 13.00 @C-3151, Pavillon Lionel-Groulx > > Ian Goodfellow will defend his thesis. The external examiner is Prof. Yann > LeCun of New York University. > > - On Wednesday, 3 Sep 2014 from 12.30 @AA3195 > > Razvan Pascanu will defend his thesis. The external examiner is Dr. Ilya > Sutskever of Google. > > I hope to see many of you there! > - Cho > >

1 0

Tea Talk 28 Aug Thursday (TOMORROW!) @13.30 AA3195 by Jeremie Zumer
by Kyung Hyun Cho 27 Aoû '14

27 Aoû '14

Dear all, As announced earlier, Jeremie Zumer will tell us about the human action recognition using deep learning tomorrow at the usual place. See below for the details. Best, - Cho === Speaker: Jeremie Zumer Date and Time: 28 Aug 2014 @13.30 Place: AA3195 Title: Deep Learning in Human Action Recognition Abstract: Hand-crafted features currently achieve state-of-the-art performance in human action recognition. As an undergraduate intern, I worked with Aaron Courville's team and investigated the application of neural-network-based models and deep architectures to this problem. During this talk, I will present some of the results of this project. >

1 0

[UPDATED!!] Tea Talk 28 Aug Thursday @13.30 AA3195 by Jeremie Zumer
by Kyung Hyun Cho 25 Aoû '14

25 Aoû '14

*The teatalk this week will be on Thursday* not on Wednesday. I apologize for my mistake in the last email. - Cho On Mon, Aug 25, 2014 at 10:21 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > This Wednesday Jeremie Zumer will tell us about the human action > recognition using deep learning. > > Best, > - Cho > > === > Speaker: Jeremie Zumer > Date and Time: 25 Aug 2014 @13.30 > Place: AA3195 > Title: Deep Learning in Human Action Recognition > Abstract: > Hand-crafted features currently achieve state-of-the-art performance in > human action recognition. As an undergraduate intern, I worked with Aaron > Courville's team and investigated the application of neural-network-based > models and deep architectures to this problem. During this talk, I will > present some of the results of this project. >

1 0

Thesis Defences: Ian Goodfellow (2 Sep) and Razvan Pascanu (3 Sep)
by Kyung Hyun Cho 25 Aoû '14

25 Aoû '14

Dear all, I'd like announce two thesis defences scheduled next week. - On Tuesday, 2 Sep 2014 from 13.00 @C-3151, Pavillon Lionel-Groulx Ian Goodfellow will defend his thesis. The external examiner is Prof. Yann LeCun of New York University. - On Wednesday, 3 Sep 2014 from 12.30 @AA3195 Razvan Pascanu will defend his thesis. The external examiner is Dr. Ilya Sutskever of Google. I hope to see many of you there! - Cho

1 1

Tea Talk 25 Aug Wednesday @13.30 AA3195 by Jeremie Zumer
by Kyung Hyun Cho 25 Aoû '14

25 Aoû '14

Dear all, This Wednesday Jeremie Zumer will tell us about the human action recognition using deep learning. Best, - Cho === Speaker: Jeremie Zumer Date and Time: 25 Aug 2014 @13.30 Place: AA3195 Title: Deep Learning in Human Action Recognition Abstract: Hand-crafted features currently achieve state-of-the-art performance in human action recognition. As an undergraduate intern, I worked with Aaron Courville's team and investigated the application of neural-network-based models and deep architectures to this problem. During this talk, I will present some of the results of this project.

1 0

Tea Talk 20 Aug Wednesday (Tomorrow!) @13.30 AA3195 by Prof. Mark Schmidt
by Kyung Hyun Cho 20 Aoû '14

20 Aoû '14

Dear all, We have a talk by Prof. Mark Schmidt from University of British Columbia (UBC). He will tell us about 'stochastic average gradient' method. I apologize for a late announcement, but hope to see many of you tomorrow! - Cho === - Speaker: Prof. Mark Schmidt (UBC) - Date and Time: 20 Aug @13.30 - Place: AA3195 - Title: Stochastic Average Gradient - Abstract: We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. Specifically, under standard assumptions the convergence rate is improved from [image: O(\frac{1}{k})] to a linear convergence rate of the form [image: O(p^k)] for some [image: p < 1]. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. Beyond these theoretical results, the algorithm also has a variety of appealing practical properties: it supports regularization and sparse datasets, it allows an adaptive step-size and has a termination criterion, it allows mini-batches, and its performance can be further improved by non-uniform sampling. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.

2 1

Tea Talk 7 Aug Friday @13.30 AA6214 by Amjad Almahairi
by Kyung Hyun Cho 08 Aoû '14

08 Aoû '14

Dear all, We have a tea talk tomorrow by Amjad Almahairi. This is a practice talk for his presentation at the CIFAR summer school next week. He will tell us about using autoencoding variational bayes for training a generative model on sentences. Hope to see many of you there! - Cho - Speaker: Amjad Almahairi - Date and Time: 7 Aug 2014 @13.30 - Place: AA6214 - Title: Variational autoencoders for language modelling - Abstract: In this talk I will talk about my recent work on the variational autoencoder (VAE) model for modelling full sentences. More specifically, I will discuss some results a variation of the VAE using the gated recursive convolutional network introduced recently by Kyung Hyun.

1 1

Tea Talk 6 Aug Wednesday @13.00 AA3195 by Dr. Peter Sunehag
by Kyung Hyun Cho 06 Aoû '14

06 Aoû '14

Dear all, We will have a talk this Wednesday by Dr. Peter Sunehag. See below for the details and the attached paper Hope to see many of you there! - Cho - Speaker: Dr. Peter Sunehag (Australian National University, http://people.cecs.anu.edu.au/user/1446) - Date and Time: 6 Aug 2014 @13.00 - Place: AA3195 - Title: Optimism as a fundamental decision-making principle - Abstract: We discuss the usefulness and rationality of optimism in general sequential decision making. Humans are often optimistically biased and optimists are achieving more of their ambitions than more rational people. We provide a mathematical analysis of general optimistic agents and identify how they can be better or worse than strictly rational agents. These agents select the most optimistic among the still plausible hypotheses from a class. Further, we discuss a milder form of optimism in the case of continuously parameterized classes. We refer to this setting as reward-modulated inference, a framework that includes recent models for synaptic weight updates in neuroscience as a special case. At the center is a trade-off between assigning high probability to likable or likely events. A nice consequence is that we can formulate generative and discriminative learning as endpoints of a continuum. Finally, I present some first experiments using autoencoders for the classical MNIST task showing a very simple way of getting ok but not state-of-the art accuracy. The aim is not to optimize narrow tasks maximally but to be able to achieve the law of effect, making choices with a frequency proportional to past rewards in situations deemed similar. In really complex tasks this can be a sufficient objective and one that humans and animals often settle for. Also, the strategy provides suitable exploration as well as robustness to change. >

1 1

Tea Talk 1 Aug Friday @13.00 AA3195 by Yoshua Bengio
by Kyung Hyun Cho 02 Aoû '14

02 Aoû '14

Dear all, We will have a tea talk this Friday by Prof. Yoshua Bengio. See below for the details and the attached paper. Hope to see many of you there! - Cho === - Speaker: Prof. Yoshua Bengio (University of Montreal) - Date and Time: 1 Aug 2014 @13.00 - Place: AA3195 - Title: How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation - Abstract: In this paper we propose to exploit reconstruction as a layer-local training signal for deep learning, be it generative or discriminant, single or multi-modal, supervised, semi-supervised or unsupervised, feedforward or recurrent. Reconstructions can be propagated in a form of target propagation playing a role similar to back-propagation but helping to reduce the reliance on back-propagation in order to perform credit assignment across many levels of possibly strong non-linearities (which is difficult for back-propagation). A regularized auto-encoder tends produce a reconstruction that is a more likely version of its input, i.e., a small move in the direction of higher likelihood. By generalizing gradients, target propagation may also allow to train deep networks with discrete hidden units. If the auto-encoder takes both a representation of input and target (or of any side information) in input, then its reconstruction of input representation provides a target towards a representation that is more likely, conditioned on all the side information. A deep auto-encoder decoding path generalizes gradient propagation in a learned way that can thus handle not just infinitesimal changes but larger, discrete changes, hopefully allowing credit assignment through a long chain of non-linear operations. For this to work, each layer must be a good denoising or regularized auto-encoder itself. In addition to each layer being a good auto-encoder, the encoder also learns to please the upper layers by transforming the data into a space where it is easier to model by them, flattening manifolds and disentangling factors. The motivations and theoretical justifications for this approach are laid down in this paper, along with conjectures that will have to be verified either mathematically or experimentally.

5 8

Re: [Lisa_teatalk] [Lisa_labo] Tea Talk 1 Aug Friday @13.00 AA3195 by Yoshua Bengio
by Kyung Hyun Cho 01 Aoû '14

01 Aoû '14

Great! I will be at the lecture hall 15 minutes prior to the tea talk. If anyone brings the recording equipment, please, stop by and test the setting with me! On Fri, Aug 1, 2014 at 11:51 AM, Bing Xu <antinucleon(a)gmail.com> wrote: > I will take my camera. It is able to record 1.5 hour in 1080i. > On Aug 1, 2014 11:49 AM, "Guillaume Alain" < > guillaume.alain.umontreal(a)gmail.com> wrote: > >> One easy option is to have the speaker wear the kind of >> microphone/headphones combination that people use to talk on their phone. >> Then you can use any phone's voice recording function to record the audio >> and it should do a good job. >> >> >> On Fri, Aug 1, 2014 at 11:45 AM, Sina Honari <sina.honari(a)gmail.com> >> wrote: >> >>> another mobile or a laptop can be used next to Yoshua for just >>> recording the voice. If the camera is away the voice shouldn't be that >>> good. >>> >>> On 1 August 2014 11:38, Tae-Ho Kim <ktho894(a)gmail.com> wrote: >>> > I'll also bring my camera as well. I've never tried recording for long >>> time, >>> > i hope that it works. >>> > >>> > On Aug 1, 2014 11:26 AM, "Kyung Hyun Cho" <cho.k.hyun(a)gmail.com> >>> wrote: >>> >> >>> >> Tapani will record the talk with his laptop. It's not going to of >>> super >>> >> quality, but hopefully will be good enough to hear and see the talk. >>> >> >>> >> >>> >> On Fri, Aug 1, 2014 at 10:06 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com >>> > >>> >> wrote: >>> >>> >>> >>> Unfortunately, I don't have anything to record the talk with. Is >>> there >>> >>> anyone else at the lab who has happened to bring their camcorder or >>> camera >>> >>> that can record the talk? >>> >>> >>> >>> >>> >>> On Fri, Aug 1, 2014 at 9:51 AM, KyoungGu Woo <epigramwoo(a)gmail.com> >>> >>> wrote: >>> >>>> >>> >>>> That's a nice idea. >>> >>>> I would also be benefited a lot. >>> >>>> >>> >>>> Kyoung-Gu >>> >>>> >>> >>>> 2014. 8. 1. 오전 9:37에 "Pierre Luc Carrier" >>> >>>> <carrier.pierreluc(a)gmail.com>님이 작성: >>> >>>> >>> >>>>> I cannot make it but I would be very interested in seeing this >>> >>>>> tea-talk. If others are in the same situation, perhaps we could >>> look into >>> >>>>> recording this tea-talk like we did with guillaume's. I do not >>> know if we >>> >>>>> have the material to do this in the lab, though. >>> >>>>> >>> >>>>> Pierre Luc >>> >>>>> >>> >>>>> >>> >>>>> 2014-07-31 17:11 GMT-04:00 Yoshua Bengio <yoshua.bengio(a)gmail.com >>> >: >>> >>>>>> >>> >>>>>> Please ignore the attached pdf, it is a very old version. The >>> arxiv >>> >>>>>> version is much better, with many mistakes fixed: >>> >>>>>> >>> >>>>>> http://arxiv.org/abs/1407.7906 >>> >>>>>> >>> >>>>>> In the future, of course, the svn version will always be the >>> latest >>> >>>>>> one (articles/2014/targetprop). >>> >>>>>> >>> >>>>>> -- Yoshua >>> >>>>>> >>> >>>>>> >>> >>>>>> On Thu, Jul 31, 2014 at 4:49 PM, Kyung Hyun Cho < >>> cho.k.hyun(a)gmail.com> >>> >>>>>> wrote: >>> >>>>>>> >>> >>>>>>> Dear all, >>> >>>>>>> >>> >>>>>>> We will have a tea talk this Friday by Prof. Yoshua Bengio. See >>> below >>> >>>>>>> for the details and the attached paper. >>> >>>>>>> >>> >>>>>>> Hope to see many of you there! >>> >>>>>>> - Cho >>> >>>>>>> >>> >>>>>>> === >>> >>>>>>> - Speaker: Prof. Yoshua Bengio (University of Montreal) >>> >>>>>>> - Date and Time: 1 Aug 2014 @13.00 >>> >>>>>>> - Place: AA3195 >>> >>>>>>> - Title: How Auto-Encoders Could Provide Credit Assignment in >>> Deep >>> >>>>>>> Networks via Target Propagation >>> >>>>>>> - Abstract: >>> >>>>>>> In this paper we propose to exploit reconstruction as a >>> layer-local >>> >>>>>>> training signal for deep learning, be it generative or >>> discriminant, single >>> >>>>>>> or multi-modal, supervised, semi-supervised or unsupervised, >>> feedforward or >>> >>>>>>> recurrent. Reconstructions can be propagated in a form of target >>> propagation >>> >>>>>>> playing a role similar to back-propagation but helping to reduce >>> the >>> >>>>>>> reliance on back-propagation in order to perform credit >>> assignment across >>> >>>>>>> many levels of possibly strong non-linearities (which is >>> difficult for >>> >>>>>>> back-propagation). A regularized auto-encoder tends produce a >>> reconstruction >>> >>>>>>> that is a more likely version of its input, i.e., a small move >>> in the >>> >>>>>>> direction of higher likelihood. By generalizing gradients, target >>> >>>>>>> propagation may also allow to train deep networks with discrete >>> hidden >>> >>>>>>> units. If the auto-encoder takes both a representation of input >>> and target >>> >>>>>>> (or of any side information) in input, then its reconstruction >>> of input >>> >>>>>>> representation provides a target towards a representation that >>> is more >>> >>>>>>> likely, conditioned on all the side information. A deep >>> auto-encoder >>> >>>>>>> decoding path generalizes gradient propagation in a learned way >>> that can >>> >>>>>>> thus handle not just infinitesimal changes but larger, discrete >>> changes, >>> >>>>>>> hopefully allowing credit assignment through a long chain of >>> non-linear >>> >>>>>>> operations. For this to work, each layer must be a good >>> denoising or >>> >>>>>>> regularized auto-encoder itself. In addition to each layer being >>> a good >>> >>>>>>> auto-encoder, the encoder also learns to please the upper layers >>> by >>> >>>>>>> transforming the data into a space where it is easier to model >>> by them, >>> >>>>>>> flattening manifolds and disentangling factors. The motivations >>> and >>> >>>>>>> theoretical justifications for this approach are laid down in >>> this paper, >>> >>>>>>> along with conjectures that will have to be verified either >>> mathematically >>> >>>>>>> or experimentally. >>> >>>>>>> >>> >>>>>>> _______________________________________________ >>> >>>>>>> Lisa_labo mailing list >>> >>>>>>> Lisa_labo(a)iro.umontreal.ca >>> >>>>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>> >>>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> _______________________________________________ >>> >>>>>> Lisa_labo mailing list >>> >>>>>> Lisa_labo(a)iro.umontreal.ca >>> >>>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>>> _______________________________________________ >>> >>>>> Lisa_labo mailing list >>> >>>>> Lisa_labo(a)iro.umontreal.ca >>> >>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>> >>>>> >>> >>>> >>> >>>> _______________________________________________ >>> >>>> Lisa_labo mailing list >>> >>>> Lisa_labo(a)iro.umontreal.ca >>> >>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>> >>>> >>> >>> >>> >> >>> >> >>> >> _______________________________________________ >>> >> Lisa_labo mailing list >>> >> Lisa_labo(a)iro.umontreal.ca >>> >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>> >> >>> > >>> > _______________________________________________ >>> > Lisa_labo mailing list >>> > Lisa_labo(a)iro.umontreal.ca >>> > https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>> > >>> _______________________________________________ >>> Lisa_labo mailing list >>> Lisa_labo(a)iro.umontreal.ca >>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>> >> >> >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >> >> > _______________________________________________ > Lisa_labo mailing list > Lisa_labo(a)iro.umontreal.ca > https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo > >

1 0

Re: [Lisa_teatalk] [Lisa_labo] Tea Talk 1 Aug Friday @13.00 AA3195 by Yoshua Bengio
by Sina Honari 01 Aoû '14

01 Aoû '14

another mobile or a laptop can be used next to Yoshua for just recording the voice. If the camera is away the voice shouldn't be that good. On 1 August 2014 11:38, Tae-Ho Kim <ktho894(a)gmail.com> wrote: > I'll also bring my camera as well. I've never tried recording for long time, > i hope that it works. > > On Aug 1, 2014 11:26 AM, "Kyung Hyun Cho" <cho.k.hyun(a)gmail.com> wrote: >> >> Tapani will record the talk with his laptop. It's not going to of super >> quality, but hopefully will be good enough to hear and see the talk. >> >> >> On Fri, Aug 1, 2014 at 10:06 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> >> wrote: >>> >>> Unfortunately, I don't have anything to record the talk with. Is there >>> anyone else at the lab who has happened to bring their camcorder or camera >>> that can record the talk? >>> >>> >>> On Fri, Aug 1, 2014 at 9:51 AM, KyoungGu Woo <epigramwoo(a)gmail.com> >>> wrote: >>>> >>>> That's a nice idea. >>>> I would also be benefited a lot. >>>> >>>> Kyoung-Gu >>>> >>>> 2014. 8. 1. 오전 9:37에 "Pierre Luc Carrier" >>>> <carrier.pierreluc(a)gmail.com>님이 작성: >>>> >>>>> I cannot make it but I would be very interested in seeing this >>>>> tea-talk. If others are in the same situation, perhaps we could look into >>>>> recording this tea-talk like we did with guillaume's. I do not know if we >>>>> have the material to do this in the lab, though. >>>>> >>>>> Pierre Luc >>>>> >>>>> >>>>> 2014-07-31 17:11 GMT-04:00 Yoshua Bengio <yoshua.bengio(a)gmail.com>: >>>>>> >>>>>> Please ignore the attached pdf, it is a very old version. The arxiv >>>>>> version is much better, with many mistakes fixed: >>>>>> >>>>>> http://arxiv.org/abs/1407.7906 >>>>>> >>>>>> In the future, of course, the svn version will always be the latest >>>>>> one (articles/2014/targetprop). >>>>>> >>>>>> -- Yoshua >>>>>> >>>>>> >>>>>> On Thu, Jul 31, 2014 at 4:49 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> We will have a tea talk this Friday by Prof. Yoshua Bengio. See below >>>>>>> for the details and the attached paper. >>>>>>> >>>>>>> Hope to see many of you there! >>>>>>> - Cho >>>>>>> >>>>>>> === >>>>>>> - Speaker: Prof. Yoshua Bengio (University of Montreal) >>>>>>> - Date and Time: 1 Aug 2014 @13.00 >>>>>>> - Place: AA3195 >>>>>>> - Title: How Auto-Encoders Could Provide Credit Assignment in Deep >>>>>>> Networks via Target Propagation >>>>>>> - Abstract: >>>>>>> In this paper we propose to exploit reconstruction as a layer-local >>>>>>> training signal for deep learning, be it generative or discriminant, single >>>>>>> or multi-modal, supervised, semi-supervised or unsupervised, feedforward or >>>>>>> recurrent. Reconstructions can be propagated in a form of target propagation >>>>>>> playing a role similar to back-propagation but helping to reduce the >>>>>>> reliance on back-propagation in order to perform credit assignment across >>>>>>> many levels of possibly strong non-linearities (which is difficult for >>>>>>> back-propagation). A regularized auto-encoder tends produce a reconstruction >>>>>>> that is a more likely version of its input, i.e., a small move in the >>>>>>> direction of higher likelihood. By generalizing gradients, target >>>>>>> propagation may also allow to train deep networks with discrete hidden >>>>>>> units. If the auto-encoder takes both a representation of input and target >>>>>>> (or of any side information) in input, then its reconstruction of input >>>>>>> representation provides a target towards a representation that is more >>>>>>> likely, conditioned on all the side information. A deep auto-encoder >>>>>>> decoding path generalizes gradient propagation in a learned way that can >>>>>>> thus handle not just infinitesimal changes but larger, discrete changes, >>>>>>> hopefully allowing credit assignment through a long chain of non-linear >>>>>>> operations. For this to work, each layer must be a good denoising or >>>>>>> regularized auto-encoder itself. In addition to each layer being a good >>>>>>> auto-encoder, the encoder also learns to please the upper layers by >>>>>>> transforming the data into a space where it is easier to model by them, >>>>>>> flattening manifolds and disentangling factors. The motivations and >>>>>>> theoretical justifications for this approach are laid down in this paper, >>>>>>> along with conjectures that will have to be verified either mathematically >>>>>>> or experimentally. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Lisa_labo mailing list >>>>>>> Lisa_labo(a)iro.umontreal.ca >>>>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Lisa_labo mailing list >>>>>> Lisa_labo(a)iro.umontreal.ca >>>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Lisa_labo mailing list >>>>> Lisa_labo(a)iro.umontreal.ca >>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>>>> >>>> >>>> _______________________________________________ >>>> Lisa_labo mailing list >>>> Lisa_labo(a)iro.umontreal.ca >>>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >>>> >>> >> >> >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >> > > _______________________________________________ > Lisa_labo mailing list > Lisa_labo(a)iro.umontreal.ca > https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >

2 1

Two tea talks this week: Coline Devin on Wednesday and Yoshua Bengio on Friday
by Kyung Hyun Cho 29 Jul '14

29 Jul '14

Dear all, I have some last minute changes to the schedule of tea talks this week: (1) Colin Devin's Tea Talk on Wednesday at 13.00 (AA3195) (2) Yoshua Bengio's Tea Talk on Friday at 13.00 (AA3195) The first one is the already announced one, and there is no change. The second one has been scheduled this morning, and Yoshua will tell us about his new idea on a new learning criterion for deep neural networks. I'm attaching the abstract by Yoshua at the end of this email. Best, - Cho === Speaker: Prof. Yoshua Bengio (University of Montreal) Title: How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation Abstract: We discuss the usefulness and rationality of optimism in general sequential decision making. Humans are often optimistically biased and optimists are achieving more of their ambitions than more rational people. We provide a mathematical analysis of general optimistic agents and identify how they can be better or worse than strictly rational agents. These agents select the most optimistic among the still plausible hypotheses from a class. Further, we discuss a milder form of optimism in the case of continuously parameterized classes. We refer to this setting as reward-modulated inference, a framework that includes recent models for synaptic weight updates in neuroscience as a special case. At the center is a trade-off between assigning high probability to likable or likely events. A nice consequence is that we can formulate generative and discriminative learning as endpoints of a continuum. Finally, I present some first experiments using autoencoders for the classical MNIST task showing a very simple way of getting ok but not state-of-the art accuracy. The aim is not to optimize narrow tasks maximally but to be able to achieve the law of effect, making choices with a frequency proportional to past rewards in situations deemed similar. In really complex tasks this can be a sufficient objective and one that humans and animals often settle for. Also, the strategy provides suitable exploration as well as robustness to change.

1 1

Tea Talk 30 July Wednesday @13.00 AA3195 by Coline Devine
by Kyung Hyun Cho 29 Jul '14

29 Jul '14

Dear all, This Wednesday at 13.00, Coline will tell us about using a neural network to learn continuous space representations of words. Hope to see many of you on Wednesday! - Cho === - Speaker: Coline Devine (Harvey Mudd College) - Date and Time: 30 July 2014 @13.00 - Place: AA3195 - Title: Learning continuous space representations of words

1 0

Tea Talk 25 July Friday @13.00 AA3195 by Bart van Merrienboer
by Kyung Hyun Cho 23 Jul '14

23 Jul '14

Dear all, This Friday at 13.00, Bart will tell us about the statistical machine translation, especially, focusing on the statistical phrase-based machine translation and how neural networks can be integrated into the existing system. >From this week, we're back at our usual teatalk place (AA3195). Hope to see many of you on Friday! - Cho ==== - Speaker: Bart van Merrienboer (University of Warwick, École Polytechnique, Chalmers Institute of Technology, University of Montreal) - Date and Time: 25 July 2014 @13.00 - Place: AA3195 - Title: Statistical Machine Translation and Neural Networks

1 0

Tea Talk 13 July Friday @13.00 Z-209 by Dustin Webb
by Kyung Hyun Cho 16 Jul '14

16 Jul '14

Hi all, This is a reminder for the upcoming tea talk at 13.00 on Friday. *The tea talk (Z-209)* will be followed by *Nicolas Boulanger's Ph.D. defence at AA-3195* at 14.00. I encourage people to attend both the tea talk and the defence. Best, - Cho On Mon, Jul 14, 2014 at 1:50 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > This Friday at 13.00, Dustin will give a tea talk about building a > deep/multilayer kernel based on the paper <Kernel Methods for Deep > Learning> by Youngmin Cho and Lawrence K. Saul presented earlier at NIPS > 2009. > > Note that the place is not the usual lecture hall in the same building, > but is Z-209 in the neighbouring building (Pavillon Claire-McNicoll). > > Hope to see many of you on Friday! > - Cho > > ==== > > - Speaker: Dustin Webb (University of Utah) > - Date and Time: 13 July 2014 @13.00 > - Place: Z-209 > - Abstract: > > We introduce a new family of positive-deﬁnite kernel functions that mimic > the > computation in large, multilayer neural nets. These kernel functions can > be used > in shallow architectures, such as support vector machines (SVMs), or in > deep > kernel-based architectures that we call multilayer kernel machines (MKMs). > We > evaluate SVMs and MKMs with these kernel functions on problems designed to > illustrate the advantages of deep architectures. On several problems, we > obtain > better results than previous, leading benchmarks from both SVMs with > Gaussian > kernels as well as deep belief nets. > > http://cseweb.ucsd.edu/~saul/papers/nips09_kernel.pdf >

3 2

Tea Talk 13 July Friday @13.00 Z-209 by Dustin Webb
by Kyung Hyun Cho 14 Jul '14

14 Jul '14

Dear all, This Friday at 13.00, Dustin will give a tea talk about building a deep/multilayer kernel based on the paper <Kernel Methods for Deep Learning> by Youngmin Cho and Lawrence K. Saul presented earlier at NIPS 2009. Note that the place is not the usual lecture hall in the same building, but is Z-209 in the neighbouring building (Pavillon Claire-McNicoll). Hope to see many of you on Friday! - Cho ==== - Speaker: Dustin Webb (University of Utah) - Date and Time: 13 July 2014 @13.00 - Place: Z-209 - Abstract: We introduce a new family of positive-deﬁnite kernel functions that mimic the computation in large, multilayer neural nets. These kernel functions can be used in shallow architectures, such as support vector machines (SVMs), or in deep kernel-based architectures that we call multilayer kernel machines (MKMs). We evaluate SVMs and MKMs with these kernel functions on problems designed to illustrate the advantages of deep architectures. On several problems, we obtain better results than previous, leading benchmarks from both SVMs with Gaussian kernels as well as deep belief nets. http://cseweb.ucsd.edu/~saul/papers/nips09_kernel.pdf

3 2

(Date Fixed) Tea Talk 18 July Friday @13.00 Z-209 by Dustin Webb
by Kyung Hyun Cho 14 Jul '14

14 Jul '14

Sorry, everyone! It's not 13 July but "18 July"! On Mon, Jul 14, 2014 at 1:50 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > This Friday at 13.00, Dustin will give a tea talk about building a > deep/multilayer kernel based on the paper <Kernel Methods for Deep > Learning> by Youngmin Cho and Lawrence K. Saul presented earlier at NIPS > 2009. > > Note that the place is not the usual lecture hall in the same building, > but is Z-209 in the neighbouring building (Pavillon Claire-McNicoll). > > Hope to see many of you on Friday! > - Cho > > ==== > > - Speaker: Dustin Webb (University of Utah) > - Date and Time: 13 July 2014 @13.00 > - Place: Z-209 > - Abstract: > > We introduce a new family of positive-deﬁnite kernel functions that mimic > the > computation in large, multilayer neural nets. These kernel functions can > be used > in shallow architectures, such as support vector machines (SVMs), or in > deep > kernel-based architectures that we call multilayer kernel machines (MKMs). > We > evaluate SVMs and MKMs with these kernel functions on problems designed to > illustrate the advantages of deep architectures. On several problems, we > obtain > better results than previous, leading benchmarks from both SVMs with > Gaussian > kernels as well as deep belief nets. > > http://cseweb.ucsd.edu/~saul/papers/nips09_kernel.pdf >

1 0

Tea Talk Today at 1pm
by Jean Pouget-Abadie 16 Jui '14

16 Jui '14

Dear all, Just a reminder that there is a tea talk at 1pm in the usual conference room. Ian Goodfellow will present his recent NIPS submission: "Adversarial sampling, a GSN killer?" AKA "Battle of the Neural Nets" Hope to see many of you there, -Jean

1 0

Re: [Lisa_teatalk] [Lisa_labo] Tea Talk 16 June Monday @13.00 AA3195 by Ian Goodfellow
by Kyung Hyun Cho 12 Jui '14

12 Jui '14

Dustin, thanks for volunteering! I've put the paper in the list of tea talk papers ( https://docs.google.com/spreadsheets/d/1CdCx2P4QXrU3byj5p3RuFw06qTLEIcchhni…) If there's no volunteer in a couple of weeks (during the tea-talk break), I will schedule Dustin in the next slot. On Thu, Jun 12, 2014 at 1:16 PM, Dustin Webb <u0625930(a)utah.edu> wrote: > I would like to give a talk but I don't have a topic per se. I encountered > a paper that attempts to do deep learning with kernel methods ( > cseweb.ucsd.edu/~yoc002/paper/nips09_arccos.pdf) if no one can offer a > better option. > > > On Thu, Jun 12, 2014 at 11:31 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> > wrote: > >> Dear all, >> >> Ian Goodfellow will guide us to the wonderful world of adversarial >> sampling away from the hell of MCMC and iterative inference at the tea talk >> on Monday (http://arxiv-web3.library.cornell.edu/abs/1406.2661, the >> paper attached) >> >> Hope to see many of you there! >> - Cho >> >> P.S. I am looking for volunteers who want to share their ideas or discuss >> interesting papers at a tea talk! See >> https://docs.google.com/spreadsheets/d/1_bbHxcm4r-rs63chcKfHuBsM3roHuud4CJJ… >> >> ======== >> >> - Speaker: Ian Goodfellow >> - Date and Time: 16 June 2014 @13.00 >> - Place: AA3195 >> - Abstract: >> >> We propose a new framework for estimating generative models via an >> adversarial process, in which we simultaneously train two models: a >> generative model G that captures the data distribution, and a >> discriminative model D that estimates the probability that a sample came >> from the training data rather than G. The training procedure for G is to >> maximize the probability of D making a mistake. This framework corresponds >> to a minimax two-player game. In the space of arbitrary functions G and D, >> a unique solution exists, with G recovering the training data distribution >> and D equal to 1/2 everywhere. In the case where G and D are defined by >> multilayer perceptrons, the entire system can be trained with >> backpropagation. There is no need for any Markov chains or unrolled >> approximate inference networks during either training or generation of >> samples. Experiments demonstrate the potential of the framework through >> qualitative and quantitative evaluation of the generated samples. >> >> >> >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >> >> > > > -- > Dustin Webb <http://www.cs.utah.edu/~dustin> > Algorithmic Robotics Lab <http://arl.cs.utah.edu/> > Laboratoire d'Informatique des Systemes Adaptatifs > <http://lisa.iro.umontreal.ca/> > Defining Intelligence <http://daemonmaker.blogspot.com> >

1 0

Tea Talk 16 June Monday @13.00 AA3195 by Ian Goodfellow
by Kyung Hyun Cho 12 Jui '14

12 Jui '14

Dear all, Ian Goodfellow will guide us to the wonderful world of adversarial sampling away from the hell of MCMC and iterative inference at the tea talk on Monday (http://arxiv-web3.library.cornell.edu/abs/1406.2661, the paper attached) Hope to see many of you there! - Cho P.S. I am looking for volunteers who want to share their ideas or discuss interesting papers at a tea talk! See https://docs.google.com/spreadsheets/d/1_bbHxcm4r-rs63chcKfHuBsM3roHuud4CJJ… ======== - Speaker: Ian Goodfellow - Date and Time: 16 June 2014 @13.00 - Place: AA3195 - Abstract: We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

1 0

Tea Talk 11 June Tomorrow @13:00 AA3195 by Prof. Tapani Raiko
by Kyung Hyun Cho 10 Jui '14

10 Jui '14

Dear all, Prof. Tapani Raiko will introduce his recent work on extending NADE into NADE-k tomorrow at the tea talk (http://arxiv.org/abs/1406.1485, the paper attached). ======= - Speaker: Prof. Tapani Raiko (Aalto University, Finland) - Date and Time: 11 June 2014 @ 13.00 - Place: AA3195 - Abstract: Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-predictive training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k has state-of-the-art performance in density estimation on the two datasets tested.

1 0

Tea Talk 9 June @13:00 AA3195 by Tae-ho Kim
by Kyung Hyun Cho 09 Jui '14

09 Jui '14

Dear all, Tae-ho Kim will tell us about the following paper: Karol Gregor and Yann LeCun. Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields. 2010. arXiv:1006.0448 [cs.NE]. http://arxiv.org/abs/1006.0448 Hope to see many of you there! ======== - Speaker: Tae-ho Kim (KAIST) - Date and Time: 9 June 2014 @ 13.00 - Place: AA3195 - Paper to be discussed: Karol Gregor and Yann LeCun. Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields. 2010. arXiv:1006.0448 [cs.NE]. http://arxiv.org/abs/1006.0448

1 1

Re: [Lisa_teatalk] [Lisa_labo] Tea Talk 9 June @13:00 AA3195 by Tae-ho Kim
by Kyung Hyun Cho 03 Jui '14

03 Jui '14

- It is 9 June. It's my fault. I didn't realize about NIPS and scheduled one tea talk this week earlier. I had to reschedule it to the following week. So, there will be two tea talks next week, one on Monday and the other on Wednesday. - During summer, depending on the travel schedules of professors and others, I will schedule tea talks a bit flexibly. Although I will announce the coming tea talks a week in advance, I'd like to ask all to check the following page once a while for the future schedule: https://docs.google.com/spreadsheets/d/1RhDOo_FfO_n6RXW9Rm_K9GWnV9lBAE6O0Ky… On Tue, Jun 3, 2014 at 6:01 PM, xavier bouthillier < xavier.bouthillier(a)gmail.com> wrote: > 9 June or 11 June? Will it be always on Monday for the summer? > > > On Tue, Jun 3, 2014 at 5:19 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> > wrote: > >> Dear all, >> >> Tae-ho Kim will tell us about the following paper: >> >> Karol Gregor and Yann LeCun. Emergence of Complex-Like Cells in a >> Temporal Product Network with Local Receptive Fields. 2010. arXiv:1006.0448 >> [cs.NE]. http://arxiv.org/abs/1006.0448 >> >> Hope to see many of you there! >> >> ======== >> >> - Speaker: Tae-ho Kim (KAIST) >> - Date and Time: 9 June 2014 @ 13.00 >> - Place: AA3195 >> - Paper to be discussed: >> Karol Gregor and Yann LeCun. Emergence of Complex-Like Cells in a >> Temporal Product Network with Local Receptive Fields. 2010. arXiv:1006.0448 >> [cs.NE]. http://arxiv.org/abs/1006.0448 >> >> >> >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >> >> >

1 0