- Lisa_teatalk - mailman.iro.umontreal.ca

Talk by Devon Hjelm this Friday (17 Apr) at 14.00 in AA3195
by Kyung Hyun Cho 14 Avr '15

14 Avr '15

Dear all, This Friday, Devon Hjelm (University of New Mexico) will visit the lab and tell us about how deep learning has helped and will help us understand "how brain works". He will be at the lab already from early Friday until later in the afternoon. If you're interested in talking to him about his or your research, feel free to email me in advance. I will try to arrange meetings. For your info, he's used RBM, DBM and DBN and is interested in applying NICE (!), VAE and NICE. Also, he's an active user of pylearn2. Hope to see many of you there! - K === Speaker: Devon Hjelm, University of New Mexico & Mind Research Network Date/time: 17 April 2015 @14.00 Place: AA3195 Title: DEEP LEARNING FOR NEUROIMAGING Abstract: The human brain did much to influence early approaches to neural networks; in return, how has deep learning helped in our understanding of the brain? In this talk, I will cover some of the available data resources for neuroimaging, including practical challenges and goals. I will then cover some recent deep learning successes and future approaches, most notably those that improve our understanding of the data and the subjects they are collected from. Finally, I will introduce some current research directions within MRN in collaboration with LISA and the University of Toronto. Bio: Devon Hjelm has been a PhD student at the University of New Mexico and the Mind Research Network since 2013. His primary research goals include using deep learning to build generative representations of the brain.

1 0

Talk by Rajesh Ranganath (Princeton University) on Friday (10 Apr) at 14.00
by Kyung Hyun Cho 07 Avr '15

07 Avr '15

Dear all, This Friday, Rajesh Ranganath (Ph.D. student at Princeton University) will visit the lab and give a talk on his recent work on "deep exponential families". As usual, the talk will start at 14.00 in AA3195. Hope to see many of you at the talk! - K === Speaker: Rajesh Ranganath, Princeton University Date/Time: 10 Apr @14.00 Place: AA3195 Title: Deep Exponential Families Abstract: We describe deep exponential families (DEFs), a class of latent variable models that are inspired by the hidden structures used in deep neural networks. DEFs capture a hierarchy of dependencies between latent variables, and are easily generalized to many settings through exponential families. We perform inference using recent ``black box" variational inference techniques. We then evaluate various DEFs on text and combine multiple DEFs into a model for pairwise recommendation data. We demonstrate that DEFs find interesting exploratory structure in large data sets, and give better predictive performance than state-of-the-art models. Bio: Rajesh Ranganath is a PhD student in machine learning working with David Blei. His primary interests are probabilistic modeling, approximate inference, Bayesian nonparametric statistics, and their applications to medicine and biology. Previously, he worked with Andrew Ng at SAIL on deep learning and Dan Jurafsky as part of the Stanford NLP Group on detecting human social intentions from conversations. Some of his recent work involves black-box inference methods, new types of random measures, and deep stochastic models.

1 0

Talk by Sungjin Ahn on 1 Apr at 14.00
by Kyung Hyun Cho 01 Avr '15

01 Avr '15

Dear all, Sungjin Ahn of UC Irvine (http://www.ics.uci.edu/~sungjia/) will visit us this Wednesday (1 Apr). He is an expert in applying Bayesian approaches to large-scale machine learning problems (stochastic MCMC). He will tell us about "Scalable MCMC in the era of Big Data" at 14.00 in the usual place (AA3195). If you're interested in talking to Sungjin, please, email me and Sungjin. Hope to see many of you on Wednesday! - Cho === Speaker: Sungjin Ahn, UC Irvine Date/Time: 1 April @14.00 Place: AA3195 Title: Scalable MCMC in the era of Big Data Abstract: Scaling up Markov chain Monte Carlo (MCMC) methods for the web-scale data is one of the most important issues in probabilistic inference in this era. Whereas the most efficient traditional MCMC algorithms, e.g. HMC, have failed to adapt to this scale since it requires to evaluate all data points per iteration, the stochastic gradient methods in optimization have successfully shown its efficiency. In this talk, I introduce a family of MCMC algorithms based the Stochastic Gradient Langevin Dynamics, and its extension for distributed computation. The algorithms bring down the computation of MCMC to the same level as the stochastic gradient optimization by making it possible for the transition kernel to be based only on mini-batches and Gaussian noises. Also, several successful results in application to the most interesting large-scale problems such as topic modeling, matrix factorization, and neural networks, are presented. Bio: Sungjin Ahn is a Ph.D. candidate in the department of Computer Science at the University of California, Irvine. Prior to UC Irvine, he received his M.Sc. at the Korea Advanced Institute of Science and Technology (KAIST) and then worked at the Agency for Defense Development as a research scientist. During his PhD, he has worked on scalable Bayesian inference methods, in particular focusing on the stochastic gradient MCMC methods. Along with Bayesian inference methods for machine learning, his research interest also includes developing efficient computational models and learning procedures for vision and natural language information.

2 2

Re: [Lisa_teatalk] Talk by Alessandro Sordoni on Friday (27 March) at 14.00 in AA3195
by Kyung Hyun Cho 28 Mar '15

28 Mar '15

Dear all, Alessandro kindly shared his slides with us. You can find the slides at https://drive.google.com/open?id=0B16RwCMQqrtdRjJvVFdMS2RIV0k&authuser=0. Best, - K On Sat, Mar 28, 2015 at 2:45 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > Alessandro kindly shared his slides with us. The slides are attached. > > Best, > - K > > On Wed, Mar 25, 2015 at 9:49 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> > wrote: > >> Dear all, >> >> This week, Alessandro Sordoni from RALI will tell us about how a >> hierarchical recurrent neural net can be used to suggest a >> information-retrieval query based on a history of previous queries/returns. >> >> Hope to see many of you there! >> - K >> >> === >> Speaker: Alessandro Sordoni >> Date/Time: 27 March @14.00 >> Place: AA3195 >> Title: Generating Context-Aware Query Suggestions with Hierarchical >> Recurrent Neural Networks >> Abstract: >> Query suggestion consists in presenting related queries to users after >> they issue one or several queries to a web search engine. Typically, >> suggestions are computed on the basis of the most recent query only. >> Taking into account the full history of previous queries as important >> context for suggestion is crucial but challenging. We propose a >> hierarchical generative model of query sequences that allows to >> generate synthetic suggestions given an unbounded history of previous >> queries as context. Our approach is based on a novel neural network >> approach that generates two levels of sequences, each with its own >> recurrent state: sequences of words, i.e. user queries, and sequences >> of queries, i.e. user sessions. Our model outperforms several >> context-aware approaches in next query prediction setting. >> Additionally, the model is able to generate fluent recommendations >> even for long-tail queries. Our findings are corroborated by a user >> study. >> >> Short bio: >> Alessandro Sordoni is a PhD student at RALI, under the direction of >> Jian-Yun Nie. >> >> >

1 0

Talk by Alessandro Sordoni on Friday (27 March) at 14.00 in AA3195
by Kyung Hyun Cho 25 Mar '15

25 Mar '15

Dear all, This week, Alessandro Sordoni from RALI will tell us about how a hierarchical recurrent neural net can be used to suggest a information-retrieval query based on a history of previous queries/returns. Hope to see many of you there! - K === Speaker: Alessandro Sordoni Date/Time: 27 March @14.00 Place: AA3195 Title: Generating Context-Aware Query Suggestions with Hierarchical Recurrent Neural Networks Abstract: Query suggestion consists in presenting related queries to users after they issue one or several queries to a web search engine. Typically, suggestions are computed on the basis of the most recent query only. Taking into account the full history of previous queries as important context for suggestion is crucial but challenging. We propose a hierarchical generative model of query sequences that allows to generate synthetic suggestions given an unbounded history of previous queries as context. Our approach is based on a novel neural network approach that generates two levels of sequences, each with its own recurrent state: sequences of words, i.e. user queries, and sequences of queries, i.e. user sessions. Our model outperforms several context-aware approaches in next query prediction setting. Additionally, the model is able to generate fluent recommendations even for long-tail queries. Our findings are corroborated by a user study. Short bio: Alessandro Sordoni is a PhD student at RALI, under the direction of Jian-Yun Nie.

1 0

Fwd: Talk by Phil Bachman on Wednesday (18 March) at 13.30
by Kyung Hyun Cho 19 Mar '15

19 Mar '15

Dear all, Phil who gave a talk today sent us the slides and a bunch of videos showing the samples from the trained models. Best, - K ---------- Forwarded message ---------- From: Phil Bachman <phil.bachman(a)gmail.com> Date: Wed, Mar 18, 2015 at 8:24 PM Subject: Re: Talk by Phil Bachman on Wednesday (18 March) at 13.30 To: Kyung Hyun Cho <cho.k.hyun(a)gmail.com> I made some nicer video demos of my models trained on MNIST and TFD. A README.txt is provided in the archive which says the following: ===== These are results from VAEs that have been trained initially using the normal one step approach, and then fine-tuned using collaborative guidance to shape the long-term behavior of the Markov chains constructed by feeding them back into themselves. The MNIST model was trained with 24x the normal KLd penalty. The TFD models were trained with 1x and 15x the normal KLd penalty. For both datasets I generated: a video of unrolled chain behavior, a set of independent samples generated by passing the isotropic Gaussian prior through the learned conditional p(x | z), and a plot of how the learned system comprising q(z | x) and p(x | z) balances between KLd and log-likelihood. The chain videos show multiple independent runs of chains with restarts every 100 steps. All nine chains were initialized with the same random example from the training set at each restart. The MNIST behavior seems pretty good. TFD with 15x KLd regularization seems to stay near the real distribution well-enough, but lacks any real detail. TFD with 1x KLd regularization (i.e. the standard free-energy) gets weird. ==== I thought some people might find this interesting. I've attached the demos and a copy of the slides from my talk. Thanks, Phil Bachman On Sat, Mar 14, 2015 at 4:42 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > Phil Bachman from our neighbouring McGill University will tell us about > training generative models, bringing in recent advances of variational > autoencoders and approximate Bayesian computation, in the view of policy > learning. See below for the detail. > > Hope to see many of you there! > - Cho > > === > Speaker: Phil Bachman (McGill University) > Date/Time: 18 March 2015 @13.30 > Place: Z-200 > Title: Learning policies for generating data > Abstract: > We develop an approach to training generative models that draws together > several current lines of research. Our approach is based on unrolling a > variational auto-encoder into a Markov chain and shaping the chain’s > trajectories using a technique inspired by recent work in Approximate > Bayesian computation. We show that the global minimizer of the resulting > objective is achieved when the generative model reproduces the target > distribution. To allow finer control over the behavior of our models, we > add a regularization term related to techniques used for shaping policy > search in reinforcement learning. We present empirical results on the MNIST > and TFD datasets which show that our approach exceeds the state-of-art > performance in quantitative measurements and from a qualitative point of > view. >

1 0

Fwd: slides for Luca Rigazio
by Kyung Hyun Cho 16 Mar '15

16 Mar '15

Dear all, Luca Rigazio (Panasonic) would like to share his slides. Find it attached, as well as in https://drive.google.com/open?id=0B16RwCMQqrtdRGFoaWpmZG8yRnMyOXF4TlFpZDlER… . Best, - K ---------- Forwarded message ---------- From: Guillaume Alain <guillaume.alain.umontreal(a)gmail.com> Date: Mon, Mar 16, 2015 at 1:08 PM Subject: slides for Luca Rigazio To: KyungHyun Cho <drunken4mylife(a)gmail.com> Hey ! Luca just sent me his slides from his presentation and asked me if I could share them with the rest of the lab. I had a look in the tea talk document on Google Drive and you're are being listed as the person who uploaded all the other slides. So, here is the pdf file for the slides.

1 0

Slides from Luca Rigazio
by Kyung Hyun Cho 16 Mar '15

16 Mar '15

Dear all, Luca Rigazio (Panasonic) would like to share his slides. Find it attached, as well as in https://drive.google.com/open?id=0B16RwCMQqrtdRGFoaWpmZG8yRnMyOXF4TlFpZDlER… . Best, - K ---------- Forwarded message ---------- From: Guillaume Alain <guillaume.alain.umontreal(a)gmail.com> Date: Mon, Mar 16, 2015 at 1:08 PM Subject: slides for Luca Rigazio To: KyungHyun Cho <drunken4mylife(a)gmail.com> Hey ! Luca just sent me his slides from his presentation and asked me if I could share them with the rest of the lab. I had a look in the tea talk document on Google Drive and you're are being listed as the person who uploaded all the other slides. So, here is the pdf file for the slides.

1 0

Talk by Phil Bachman on Wednesday (18 March) at 13.30
by Kyung Hyun Cho 14 Mar '15

14 Mar '15

Dear all, Phil Bachman from our neighbouring McGill University will tell us about training generative models, bringing in recent advances of variational autoencoders and approximate Bayesian computation, in the view of policy learning. See below for the detail. Hope to see many of you there! - Cho === Speaker: Phil Bachman (McGill University) Date/Time: 18 March 2015 @13.30 Place: Z-200 Title: Learning policies for generating data Abstract: We develop an approach to training generative models that draws together several current lines of research. Our approach is based on unrolling a variational auto-encoder into a Markov chain and shaping the chain’s trajectories using a technique inspired by recent work in Approximate Bayesian computation. We show that the global minimizer of the resulting objective is achieved when the generative model reproduces the target distribution. To allow finer control over the behavior of our models, we add a regularization term related to techniques used for shaping policy search in reinforcement learning. We present empirical results on the MNIST and TFD datasets which show that our approach exceeds the state-of-art performance in quantitative measurements and from a qualitative point of view.

1 0

Re: [Lisa_teatalk] [Lisa_labo] Tea Talk by Kyle Kastner this Friday 13.30-14.30 @AA3195
by Kyle Kastner 11 Mar '15

11 Mar '15

Definitely! I might even send them beforehand if they are finished early :) On Wed, Mar 11, 2015 at 8:18 AM, Atousa Torabi <torabi.atousa(a)gmail.com> wrote: > Kyle, > > Could you please send us your talk slides after your talk? > > Thanks, > Atousa > > On Tue, Mar 10, 2015 at 9:21 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> > wrote: >> >> Dear all, >> >> Again, sorry about the mistake in the email body. It's "March" not >> "April". I apologize. >> >> To those who asked me about the potential change of time to 14.00: >> unfortunately, it's not possible, as there will be another event at the same >> place starting at 15.00. >> >> Best, >> - K >> >> On Tue, Mar 10, 2015 at 6:39 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> >> wrote: >>> >>> Dear all, >>> >>> Last year Kyle told us about sklearn-theano which can be used to bridge >>> between scikit-learn and Theano. Now, he's ready to tell us how we can even >>> bridge Theano with Caffe (!) In other words, we can use GoogLeNet easily >>> together with Theano now. Isn't it awesome? Furthermore, it's going to be >>> the thirteenth this Friday (huh?) >>> >>> The talk will be on Friday at 13.30 in the usual place AA3195. See below >>> for the details. >>> - Cho >>> >>> === >>> Speaker: Kyle Kaster >>> Date/Time: 13 Apr (Friday) @13.30 >>> Place: AA3195 >>> Title: Theano + Caffe, with a side of GoogLeNet, in sklearn-theano >>> Abstract: >>> sklearn-theano (http://sklearn-theano.github.io/) is a package for >>> applying pretrained neural networks to data using python, numpy and >>> theano. sklearn-theano strives to conform to the scikit-learn >>> interface, and can be flexibly added into machine learning pipelines >>> as an upfront transform followed by standard classifiers and >>> transformers from scikit-learn. Many of the models in sklearn-theano >>> also work as standalone classifiers and object localizers. >>> >>> We have recently added support for a potentially huge number of >>> pretrained models by adding a bridge between Theano and Caffe. This >>> talk will go over some interesting notes about the architecture of >>> GoogLeNet itself, the key things necessary to bring *new* models into >>> sklearn-theano (or theano in general), and some discussion about where >>> we plan to go next. >>> >>> >> >> >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >> >

1 0

Tea Talk by Kyle Kastner this Friday 13.30-14.30 @AA3195
by Kyung Hyun Cho 11 Mar '15

11 Mar '15

Dear all, Last year Kyle told us about sklearn-theano which can be used to bridge between scikit-learn and Theano. Now, he's ready to tell us how we can even bridge Theano with Caffe (!) In other words, we can use GoogLeNet easily together with Theano now. Isn't it awesome? Furthermore, it's going to be the thirteenth this Friday (huh?) The talk will be on Friday at 13.30 in the usual place AA3195. See below for the details. - Cho === Speaker: Kyle Kaster Date/Time: 13 Apr (Friday) @13.30 Place: AA3195 Title: Theano + Caffe, with a side of GoogLeNet, in sklearn-theano Abstract: sklearn-theano (http://sklearn-theano.github.io/) is a package for applying pretrained neural networks to data using python, numpy and theano. sklearn-theano strives to conform to the scikit-learn interface, and can be flexibly added into machine learning pipelines as an upfront transform followed by standard classifiers and transformers from scikit-learn. Many of the models in sklearn-theano also work as standalone classifiers and object localizers. We have recently added support for a potentially huge number of pretrained models by adding a bridge between Theano and Caffe. This talk will go over some interesting notes about the architecture of GoogLeNet itself, the key things necessary to bring *new* models into sklearn-theano (or theano in general), and some discussion about where we plan to go next.

2 2

[Updated] Tea Talk by Orhan Firat "TOMORROW" at 14.00 in AA3195
by Kyung Hyun Cho 05 Mar '15

05 Mar '15

Dear all, I'm yet again terribly sorry about the wrong date in my previous email! My internal clock is all messed up these days, and thought it was Friday already. :( It is "TOMORROW" not today. Best, - Cho On Thu, Mar 5, 2015 at 7:05 AM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > We have a tea talk today at 14.00 in our usual tea talk venue (AA3195). > Orhan Firat will tell us about how we can use deep learning to read your > mind (!) Unless you want to be a victim of mind reading, I highly suggest > all of you to attend the talk and learn how to avoid it as much as possible > (if you can at all.) > > See below for the details, and hope many of you will attend! > - Cho > > P.S. I'm terribly sorry for the late announcement. > > === > - Speaker: Orhan Firat (Middle East Technical University, Turkey) > - Date/time: TODAY 6 March 2015 @14.00 > - Place: AA3195 > - Title: Deep Learning for (Functional) Neuroimaging > - Abstract: Deep learning has made a significant impact on many fields of > research, but not contributed much to neuro-imaging, the eyes of > neuro-science. In this talk, I will focus on deep learning opportunities in > (functional) neuro-imaging. I will present the challenges in the field and > give some application examples of deep learning for brain state decoding > (mind reading), intrinsic network discovery and mental disease diagnosis. > Further I will try to discuss about the applicability on subject-transfer > decoding and some generative tasks. > > >

1 0

Tea Talk by Orhan Firat "TODAY" at 14.00 in AA3195
by Kyung Hyun Cho 05 Mar '15

05 Mar '15

Dear all, We have a tea talk today at 14.00 in our usual tea talk venue (AA3195). Orhan Firat will tell us about how we can use deep learning to read your mind (!) Unless you want to be a victim of mind reading, I highly suggest all of you to attend the talk and learn how to avoid it as much as possible (if you can at all.) See below for the details, and hope many of you will attend! - Cho P.S. I'm terribly sorry for the late announcement. === - Speaker: Orhan Firat (Middle East Technical University, Turkey) - Date/time: TODAY 6 March 2015 @14.00 - Place: AA3195 - Title: Deep Learning for (Functional) Neuroimaging - Abstract: Deep learning has made a significant impact on many fields of research, but not contributed much to neuro-imaging, the eyes of neuro-science. In this talk, I will focus on deep learning opportunities in (functional) neuro-imaging. I will present the challenges in the field and give some application examples of deep learning for brain state decoding (mind reading), intrinsic network discovery and mental disease diagnosis. Further I will try to discuss about the applicability on subject-transfer decoding and some generative tasks.

1 0

Talk by Luca Rigazio on Wednesday
by Kyung Hyun Cho 04 Mar '15

04 Mar '15

Dear all, We have a talk this Wednesday by Luca Rigazio on convolutional neural networks at 14.00. See the details below. Hope to see many of you there! - K --- Luca Rigazio - Panasonic Silicon Valley Laboratory ML background (MS, no Ph.D., sorry), I worked extensively in speech recognition/NLU, now I run PSVL in Cupertino, California, Panasonic's elite corporate innovation lab. Last year I caught the deep-learning virus (like many) and started working on it (like most), spun off a new dedicated team. We are still learning but enjoy doing cool stuff, I enjoy coding new algorithms with the team (mostly C++, don't like Matlab, starting to cope with Python). Today I will tell you a couple of things about us and our company, present some initial results especially in Vision, and hope to learn a lot from your great group! -- Deep Clustered Convolutional Kernels DNN's have recently achieved state of the art performance: in practice the network architecture has to be manually set by domain experts, generally by a costly trial and error procedure, which often accounts for a large portion of the final system performance. We view this as a limitation and propose a training algorithm that automatically optimizes network architecture, by progressively increasing model complexity and then eliminating model redundancy by selectively removing parameters at training time. For convolutional neural networks, our method relies on iterative split/merge clustering of convolutional kernels interleaved by stochastic gradient descent. We present a training algorithm and experimental results on three different vision tasks, showing improved performance compared to similarly sized hand-crafted architectures. >From my mobile

2 1

Harri Valpola's Slides
by Kyung Hyun Cho 04 Mar '15

04 Mar '15

Dear all, Harri wanted me to forward his message along with his slides: I've been pretty busy for the past weeks but I hope to be able to test whether I will manage to train RNNs using the same principles at some point. As I said, I really like your work on translation, including dynamics, attention, etc. There seem to be many important aspects that need to be mastered in other problems, too (I'm thinking of robotics in particular). I hope I will hear more about your research soon. Best regards, - Harri -

1 0

Tea Talk by Julian Serban this Friday (27 Feb) 14.00 - 15.00 @AA3195
by Kyung Hyun Cho 27 Fév '15

27 Fév '15

Dear all, Julian Serban will tell us about "Learning to Play Chess with Deep Learning" this Friday at the usual place starting from 14.00. Hope to see many of you at the tea talk! - Cho === Speaker: Julian Serban Date/Time: 27 Feb 2015 14.00-15.00 Place: AA3195 Title: Learning to Play Chess with Deep Learning Abstract: TBA

1 1

Tea Talk by Prof. Yoshua Bengio on Friday (20 Feb) at 14.00
by Kyung Hyun Cho 20 Fév '15

20 Fév '15

Dear all, We will be back to the usual tea talk schedule starting from next Friday with Yoshua introducing us a biologically plausible learning algorithm for training a deep generative neural network. The talk will start at 14.00 in the usual place of AA3195. Hope to see many of you there! - Cho === Speaker: Prof. Yoshua Bengio Date/Time: 14.00 - 15.00 on Friday (20 Feb) Place: AA3195 Title: Biologically Plausible Deep Learning Abstract: Neuroscientists have long criticised deep learning algorithms as incompatible with current knowledge of neurobiology. We explore more biologically plausible versions of deep representation learning, focusing here mostly on unsupervised learning but developing a learning mechanism that could account for supervised, unsupervised and reinforcement learning. The starting point is that the basic learning rule believed to govern synaptic weight updates (Spike-Timing Dependent Plasticity) can be interpreted as gradient descent on some objective function so long as the neuronal dynamics push firing rates towards better values of the objective function (be it supervised, unsupervised, or reward-driven). The second main idea is that this corresponds to a form of variational EM algorithm in which the above dynamics correspond to approximate inference. Another contribution of this paper is that the gradients required for updating the hidden states in the above variational interpretation can be estimated using an approximation that only requires propagating activations forward and backward, with pairs of layers learning to form a denoising auto-encoder. Finally, we extend the theory about the probabilistic interpretation of autoencoders to justify improved sampling schemes based on the generative interpretation of denoising auto-encoders, and we validate all these ideas on generative learning tasks.

2 1

Tea Talk by Prof. Yoshua Bengio on Friday (20 Feb) at 14.00
by Kyunghyun Cho 14 Fév '15

14 Fév '15

Dear all, We will be back to the usual tea talk schedule starting from next Friday with Yoshua introducing us a biologically plausible learning algorithm for training a deep generative neural network. The talk will start at 14.00 in the usual place of AA3195. Hope to see many of you there! - Cho === Speaker: Prof. Yoshua Bengio Date/Time: 14.00 - 15.00 on Friday (20 Feb) Place: AA3195 Title: Biologically Plausible Deep Learning Abstract: Neuroscientists have long criticised deep learning algorithms as incompatible with current knowledge of neurobiology. We explore more biologically plausible versions of deep representation learning, focusing here mostly on unsupervised learning but developing a learning mechanism that could account for supervised, unsupervised and reinforcement learning. The starting point is that the basic learning rule believed to govern synaptic weight updates (Spike-Timing Dependent Plasticity) can be interpreted as gradient descent on some objective function so long as the neuronal dynamics push firing rates towards better values of the objective function (be it supervised, unsupervised, or reward-driven). The second main idea is that this corresponds to a form of variational EM algorithm in which the above dynamics correspond to approximate inference. Another contribution of this paper is that the gradients required for updating the hidden states in the above variational interpretation can be estimated using an approximation that only requires propagating activations forward and backward, with pairs of layers learning to form a denoising auto-encoder. Finally, we extend the theory about the probabilistic interpretation of autoencoders to justify improved sampling schemes based on the generative interpretation of denoising auto-encoders, and we validate all these ideas on generative learning tasks.

1 0

Talk by Dr. Harri Valpola on Wednesday 11 Feb at 13.30 @AA3195
by Kyung Hyun Cho 07 Fév '15

07 Fév '15

Dear all, Next week, we will have a visitor from Finland. Dr. Harri Valpola is a machine learning researcher from ZenRobotics (Helsinki) and has been collaborating with Prof. Tapani Raiko on various topics including unsupervised learning. He will arrive at the lab in the afternoon on Tuesday (around 2pm). If anyone's interested in talking to him, please, let him know by replying to his email (harri.valpola(a)iki.fi) On the next day (Wednesday), he will tell us about his recent research on unsupervised neural networks at our usual seminar place starting at 13.30. Please, see below for Harri's message! - Cho === Title of the talk: Deep unsupervised and semi-supervised learning Speaker: Harri Valpola, ZenRobotics Ltd. I'm Harri Valpola, a machine learning researcher from Helsinki, currently working for ZenRobotics which I co-founded in 2007. Between 2010 and 2014 I was temporarily focused on product development (machine learning applied to robots) but now that our product (robotic waste sorting) is pretty much ready, I have been able to return back to research since summer 2014. My main research topic has been unsupervised learning ever since I started working in Kohonen's lab in 1993. Now I'm going to talk about a model which I've been working on in collaboration with Tapani Raiko and Antti Rasmus from Aalto University. The model combines denoising autoencoders and denoising source separation. The goal is to design an unsupervised learning method which supports supervised learning as well as possible. The network needs to be able to focus on abstract invariant features on the higher layers and learn deep hierarchies efficiently. This is achieved by shortcut connections from the encoder to the decoder and cost functions on many layers of the network (not just reconstruction of observations and prediction of targets). There are published results in two arXiv papers ( https://arxiv.org/abs/1411.7783 and https://arxiv.org/abs/1412.7210) but I'm also going to talk a bit about our very preliminary results on semi-supervised learning. I'm visiting Montreal between Tuesday and Thursday (Feb 10-12) and I hope to meet many of you. In general, I'm interested in anything that helps me in my pursue: I want to build a brain for autonomous robots. Part of the solution, I believe, is a "cortical algorithm". Currently there are good solutions for some subtasks but not for all (or even most) of them at the same time: * deep hierarchy of abstract invariant features * semi-supervised learning * efficient inference (state estimation which integrates information from all modalities and across time) * segmentation * attention * simulation and planning * representation, detection and learning of relations Perhaps there are solutions that I'm not aware of -- I hope someone gives me some tips in Montreal :-) The model I'm pursuing should be able to handle time-series data and work efficiently so optimization is also among the interesting topics. The sampling aspect of GSN is definitely interesting, too. I don't know if anybody is doing robotics-related work in Montreal but I'll mention a few topics that I'm interested in, just in case: * hierarchical control (especially unsupervised and reinforcement learning of hc) and decision making * working memory and attention (and control of them) * cerebellar learning (related to iterative learning control) In case someone is interested to hear my comments about your own work, I can advertise my own background (chronological order so starts more than 20 years back): * invariant features from temporal slowness * sparse coding * variational unsupervised learning of generative models * denoising source separation * hierarchical latent variable models * cerebellar learning * hierarchical control * higher-order models * segmentation and attention * representation, detection and learning of relations * reinforcement learning I've worked with various types of data, including: * speech * images * industrial processes * astrophysics * climate * neuroimaging * various robotics-related I'm looking forward to meeting you in Montreal, - Harri -

1 0

New Tea Talk Schedule
by Kyung Hyun Cho 23 Jan '15

23 Jan '15

Dear all, According to the survey, we have decided on a new time for tea talks: *14.00 - 15.00 Fridays* The next tea talk is scheduled a week after the ICML deadline, and I will make an announcement a week before that. Best, - K

1 0

Survey on New TeaTalk Schedule
by Kyung Hyun Cho 20 Jan '15

20 Jan '15

Dear all, As I have announced earlier, we are at the moment considering to move the tea talk slot from Wednesday to either the beginning of or the end of each week. Since the last email, I have received a couple of replies from the lab members. Considering those replies, we have narrowed down our options into 1. Fridays: 14h00 to 15h00 (preferred) 2. Mondays: 10h30 to 11h30 Please, reply to this email to express your preference among these options. I will take "no reply" as a sign of "either of them being fine". Best, - K

15 19

21 Jan 2015: Talk by Adriana Romero on Wednesday at 13.30 (AA3195)
by Kyung Hyun Cho 18 Jan '15

18 Jan '15

Dear all, Adriana Romero, a visiting researcher at the lab from Barcelona (!), will tell us about her latest work done here. She will teach us how to raise our neural net to become a leaner, but taller network. The talk will be at the usual place AA3195 starting from 13.30 on Wednesday (21 Jan). Hope to see many of you there! - Cho === - *Speaker*: Adriana Romero (University of Barcelona) - *Date/Time*: 13.30 - 14.30, 21 Jan 2015 - *Place*: AA3195 - *Title*: FitNets: Hints for Thin Deep Nets - *Abstract*: While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. We extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network. - *Bio*: I am currently a PhD student at University of Barcelona, advised by Dr. Carlo Gatta, working on deep learning models and their applications to computer vision. I graduated form Universitat Autònoma de Barcelona in 2010 as Computer Engineer and from Universitat Politècnica de Catalunya in 2012 as M.Sc. in Artificial Intelligence. My previous work was focused on unsupervised sparse feature learning algorithms to train both shallow and deep networks. In August 2014, I joined LISA lab for 6 months, where I've been working with Prof. Yoshua Bengio on training thin and deep student networks from shallower and wide teacher networks.

1 0

Introduction to Tea Talk, this Wednesday @13.30, AA3195
by Kyung Hyun Cho 13 Jan '15

13 Jan '15

Dear all, This week, we will not have any tea talk. There will be a quick introductory session for new people (including those who started last autumn) on how tea talks at the lab are arranged and what one should do to schedule her/his own tea talk. So, if you're relatively new to the lab and would like to know why there are so many interesting/boring talks at the lab and how you can be yet another interesting/boring speaker, please, come to AA3195 at 13.30 this Wednesday. Best, - Cho

1 0

Tea Talks 2015
by Kyung Hyun Cho 08 Jan '15

08 Jan '15

Dear all, With the new year begins the new series of tea talks! As usual with the last year, we will host a series of (tea) talks on Wednesdays starting from 13.30 lasting one hour at the usual place of AA3195. Again as usual, the day, time and place are subject to change. You can check the schedule of the tea talks at https://docs.google.com/spreadsheets/d/1_bbHxcm4r-rs63chcKfHuBsM3roHuud4CJJ… We already have a few talks scheduled for this semester, but obviously a lot of slots are empty now. Please, feel free to contact me if you want to give a talk. I especially encourage new visitors to tell us about themselves and their research at the tea talks. Best, - K

1 0

Re: [Lisa_teatalk] Schedule of Talks before and after NIPS 2014
by Kyung Hyun Cho 16 Déc '14

16 Déc '14

Dear all, It is the final day of our Pre-/Post- NIPS seminar series tomorrow! Those two talks will also conclude the series of talks we have had this year (2014). Both of the talks will be held at the usual place at 10.30 and at 13.30. - 10.30 - 11.30 <15 million predictions per seconds: the scientific challenges at Criteo> by Nicolas le Roux (Criteo) - 13.30 - 14.30 <General Stochastic Networks for Classification> by Matthias Zohrer (TU Graz) Hope to see many of you tomorrow! - Cho *= Schedule =* Available also at: http://lisa.iro.umontreal.ca/seminars_en.html + 3 Dec 2014 (Wed) - 13.30 - 14.30 <A Multiplicative Model for Learning Distributed Text-Based Attribute Representations> by Ryan Kiros (University of Toronto) + 4 Dec 2014 (Thu) - 10.30 - 11.30 <What if computer can perceive the world like we do?> by Xinyi Chen, Peng (Eric) Ding (DeepGlint) - 13.30 - 14.30 <Learning Compact Yet Rich Models for Text> by Sarath Chandar (IBM India) + 5 Dec 2014 (Fri) - 10.30 - 11.30 <Joint Training of Deep Autoencoders> by Dr. Yingbo Zhou (University at Buffalo, The State University of New York) - 13.30 - 14.30 <How Watson Learns Superhuman Jeopardy! Strategies> by Dr. Gerald Tesauro (IBM T. J. Watson Research Center) + 15 Dec 2014 (Mon) - 10.30 - 11.30 <On the Number of Linear Regions of Deep Neural Networks> by Dr. Guido Montufar <MPI MIS> - 13.30 - 14.30 <Machine Learning with Nearest Neighbors> by Prof. Yung-Kyun Noh (KAIST) *+ 16 Dec 2014 (Tue)* * - 10.30 - 11.30 <15 million predictions per seconds: the scientific challenges at Criteo> by Nicolas le Roux (Criteo)* * - 13.30 - 14.30 <General Stochastic Networks for Classification> by Matthias Zohrer (TU Graz)*

2 3