- Lisa_teatalk - mailman.iro.umontreal.ca

Tea Talk Tomorrow 28 May @13:00 AA3195 by Guillaume Alain
by Kyung Hyun Cho 27 Mai '14

27 Mai '14

Dear all, Guillaume Alain will tell us about the recently proposed technique of using FFT for fast convolutional operation tomorrow. See below for the detail. See you there! - Cho ============ - Speaker: Guillaume Alain - Date and Time: 28 May 2014 @ 13.00 - Place: AA3195 - Paper to be Discussed: Michael Mathieu, Mikael Henaff, Yann LeCun. Fast Training of Convolutional Networks through FFTs. ICLR 2014

6 6

Tea Talk Wed 28 May @13:00 AA3195 by Guillaume Alain
by Kyung Hyun Cho 23 Mai '14

23 Mai '14

Dear all, Guillaume Alain will talk about a recent paper <Fast Training of Convolutional Networks through FFTs <http://arxiv.org/abs/1312.5851>> presented at ICLR 2014 by Michael Mathieu, Mikael Henaff and Yann LeCun at the tea talk next week in the usual place (AA3195) at the usual time (13.00) Mark you calendar and see you there! - Cho P.S. I am looking for lab members or potential visitors who are willing to talk about either (1) their own work, (2) interesting papers from outside the lab (both old and new papers are fine) and (3) potential research ideas. Please, see the available time slots from https://docs.google.com/spreadsheets/d/1RhDOo_FfO_n6RXW9Rm_K9GWnV9lBAE6O0Ky… and email me!

2 1

Tea Talk Fri 23 May @13:00 AA3195 by Dustin Webb
by Kyung Hyun Cho 23 Mai '14

23 Mai '14

Dear all, This is a reminder for the tea talk schedule today at 13.00. See you soon! - Cho == Speaker: Dustin Webb (University of Utah) Place: AA3195 Date/Time: 23 May 2014 @13.00 - 14.00 Title: Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs Abstract: An accurate dynamics model of a robot is an important ingredient of many algorithms used to solve robotics problems, including motion planning, control, localization, and mapping. Models derived from ﬁrst principles often contain parameters (e.g. mass, moment of inertia, arm lengths, etc.) for which values are unknown. Those which cannot be easily measured must be estimated from the observed behavior of the robot. A good approach to address this problem is to plan control policies for the robot that elicit maximal amounts of information about the parameters of the system, while still achieving other objectives speciﬁed for the robot. In case of parameters subject to drift, this must be done continuously over the lifetime of the robot if costly re-calibrations are to be avoided. In this paper, we introduce a new method that formulates the parameter estimation problem as a continuous partially-observable Markov decision process (POMDP), which plans control policies that optimally trade-off the effort spent on learning parameters and effort spent on achieving regular robot objectives (exploration vs. exploitation), and allow for online, continual parameter estimation. While POMDPs have, until recently, been mostly of theoretical interest due to their inherent complexity, we build on recent advances that allow continuous, Gaussian POMDPs to be approximately-optimally solved in near-real-time rates. We show that the computed control policies lead to improved convergence of the belief of the parameters compared to system identiﬁcation approaches based on applying random controls. Dustin J. Webb, Kyle L. Crandall and Jur van den Berg. Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs. ICRA 2014. http://arl.cs.utah.edu/pubs/ICRA2014-ope.pdf

1 0

Tea talk Wed 21 May @13:00 AA3195 by Prof. Alain Tapp - Slides Attached
by Kyung Hyun Cho 22 Mai '14

22 Mai '14

Dear all, I'm forwarding the slides used yesterday by Prof. Alain Tapp during the tea talk. Best, - Cho

1 0

Tea talk Friday 23 May @13:00 AA3195 by Dustin Webb
by Kyung Hyun Cho 20 Mai '14

20 Mai '14

Dear all, I would like to announce a tea talk that will be given by Dustin Webb this Friday at 13.00. Mark your calendar and see you there! - Cho == Speaker: Dustin Webb (University of Utah) Place: AA3195 Date/Time: 23 May 2014 @13.00 - 14.00 Title: Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs Abstract: An accurate dynamics model of a robot is an important ingredient of many algorithms used to solve robotics problems, including motion planning, control, localization, and mapping. Models derived from ﬁrst principles often contain parameters (e.g. mass, moment of inertia, arm lengths, etc.) for which values are unknown. Those which cannot be easily measured must be estimated from the observed behavior of the robot. A good approach to address this problem is to plan control policies for the robot that elicit maximal amounts of information about the parameters of the system, while still achieving other objectives speciﬁed for the robot. In case of parameters subject to drift, this must be done continuously over the lifetime of the robot if costly re-calibrations are to be avoided. In this paper, we introduce a new method that formulates the parameter estimation problem as a continuous partially-observable Markov decision process (POMDP), which plans control policies that optimally trade-off the effort spent on learning parameters and effort spent on achieving regular robot objectives (exploration vs. exploitation), and allow for online, continual parameter estimation. While POMDPs have, until recently, been mostly of theoretical interest due to their inherent complexity, we build on recent advances that allow continuous, Gaussian POMDPs to be approximately-optimally solved in near-real-time rates. We show that the computed control policies lead to improved convergence of the belief of the parameters compared to system identiﬁcation approaches based on applying random controls. Dustin J. Webb, Kyle L. Crandall and Jur van den Berg. Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs. ICRA 2014. http://arl.cs.utah.edu/pubs/ICRA2014-ope.pdf

1 0

Tea talk Wed 21 May @13:00 AA3195 by Prof. Alain Tapp
by Kyung Hyun Cho 20 Mai '14

20 Mai '14

Dear all, This is a reminder for the tea talk schedule tomorrow at 13.00. See you all tomorrow! - Cho == Speaker: Prof. Alain Tapp (Département d’informatique et de recherche opérationnelle, Université de Montréal) Title: A new machine learning algorithm based on binary circuits Abstract: N/A Place: AA3195 Date/Time: 21 May 2014 @13.00 - 14.00

1 0

Tea talk Wednesday 21st May @13:00 AA3195 by Prof. Alain Tapp
by Kyung Hyun Cho 14 Mai '14

14 Mai '14

Dear all, We will have a tea talk next Wednesday by Prof. Alain Tapp. See details below. Best, - Cho == Speaker: Prof. Alain Tapp (Département d’informatique et de recherche opérationnelle, Université de Montréal) Title: A new machine learning algorithm based on binary circuits Abstract: N/A Place: AA3195 Date/Time: 21 May 2014 @13.00 - 14.00

1 0

Announcements about Tea Talks
by Kyunghyun Cho 09 Mai '14

09 Mai '14

Hi all, I will be taking over the role of arranging tea talks after Razvan graduates and leaves the lab (Congrats, Razvan!) I have a few announcements to make here. (1) If you want to give a teatalk: Please, feel free to contact me about it. As far as I understand, a topic for a teatalk may include; (a) your own work (b) others' work that may interest the members of the lab (c) some new ideas that others may have interest in I'd especially like to emphasize (b). Often, it's easier and faster to understand the concept of a paper, if someone explains it in 'spoken' words, rather than 'written' ones. This will help all the members keep up with the latest and important topics in the field. Also, if you're hosting a visitor at the lab and she/he's willing to give a talk, do let me so that I can book a time and place in advance. (2) Prof. Alain Tapp: Wednesday 21 June @13.00 AA3195 The title and abstract are not available yet, but I will send an email once more when I have them. (3) Dustin Webb: Friday 23 June @13.00 AA3195 Title: Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs Abstract: An accurate dynamics model of a robot is an important ingredient of many algorithms used to solve robotics problems, including motion planning, control, localization, and mapping. Models derived from ﬁrst principles often contain parameters (e.g. mass, moment of inertia, arm lengths, etc.) for which values are unknown. Those which cannot be easily measured must be estimated from the observed behavior of the robot. A good approach to address this problem is to plan control policies for the robot that elicit maximal amounts of information about the parameters of the system, while still achieving other objectives speciﬁed for the robot. In case of parameters subject to drift, this must be done continuously over the lifetime of the robot if costly re-calibrations are to be avoided. In this paper, we introduce a new method that formulates the parameter estimation problem as a continuous partially-observable Markov decision process (POMDP), which plans control policies that optimally trade-off the effort spent on learning parameters and effort spent on achieving regular robot objectives (exploration vs. exploitation), and allow for online, continual parameter estimation. While POMDPs have, until recently, been mostly of theoretical interest due to their inherent complexity, we build on recent advances that allow continuous, Gaussian POMDPs to be approximately-optimally solved in near-real-time rates. We show that the computed control policies lead to improved convergence of the belief of the parameters compared to system identiﬁcation approaches based on applying random controls. Dustin J. Webb, Kyle L. Crandall and Jur van den Berg. Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs. ICRA 2014. http://arl.cs.utah.edu/pubs/ICRA2014-ope.pdf

1 0

Announcements on Tea Talk
by Kyung Hyun Cho 09 Mai '14

09 Mai '14

Hi all, I will be taking over the role of arranging tea talks after Razvan graduates and leaves the lab (Congrats, Razvan!) I have a few announcements to make here. (1) If you want to give a teatalk: Please, feel free to contact me about it. As far as I understand, a topic for a teatalk may include; (a) your own work (b) others' work that may interest the members of the lab (c) some new ideas that others may have interest in I'd especially like to emphasize (b). Often, it's easier and faster to understand the concept of a paper, if someone explains it in 'spoken' words, rather than 'written' ones. This will help all the members keep up with the latest and important topics in the field. Also, if you're hosting a visitor at the lab and she/he's willing to give a talk, do let me so that I can book a time and place in advance. (2) Prof. Alain Tapp: Wednesday 21 June @13.00 AA3195 The title and abstract are not available yet, but I will send an email once more when I have them. (3) Dustin Webb: Friday 23 June @13.00 AA3195 Title: Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs Abstract: An accurate dynamics model of a robot is an important ingredient of many algorithms used to solve robotics problems, including motion planning, control, localization, and mapping. Models derived from ﬁrst principles often contain parameters (e.g. mass, moment of inertia, arm lengths, etc.) for which values are unknown. Those which cannot be easily measured must be estimated from the observed behavior of the robot. A good approach to address this problem is to plan control policies for the robot that elicit maximal amounts of information about the parameters of the system, while still achieving other objectives speciﬁed for the robot. In case of parameters subject to drift, this must be done continuously over the lifetime of the robot if costly re-calibrations are to be avoided. In this paper, we introduce a new method that formulates the parameter estimation problem as a continuous partially-observable Markov decision process (POMDP), which plans control policies that optimally trade-off the effort spent on learning parameters and effort spent on achieving regular robot objectives (exploration vs. exploitation), and allow for online, continual parameter estimation. While POMDPs have, until recently, been mostly of theoretical interest due to their inherent complexity, we build on recent advances that allow continuous, Gaussian POMDPs to be approximately-optimally solved in near-real-time rates. We show that the computed control policies lead to improved convergence of the belief of the parameters compared to system identiﬁcation approaches based on applying random controls. Dustin J. Webb, Kyle L. Crandall and Jur van den Berg. Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs. ICRA 2014. http://arl.cs.utah.edu/pubs/ICRA2014-ope.pdf

1 0

Tea talk Wednesday 9th April @13:00 AA3195 by David WF and Razvan
by Razvan Pascanu 08 Avr '14

08 Avr '14

Hi all, This Wednesday, same place AA3195, same time 13:00, we have a double tea talk featured. First David will present his (and co-authors) paper accepted at ICLR 2014. After that I will do a practice talk for my oral at ICLR as well. Be warned that this tea-talk might take longer than 1h all together. Talk by: David Warde Farley Title :An empirical analysis of dropout in piecewise linear networks Abstract: The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient. paper on openreview<http://openreview.net/document/f4c625c6-b0eb-4fd3-ab50-25182fe68733#f4c625c…> Talk by: Razvan Pascanu (practice talk of 15 +5 minutes oral presentation) Tile : Revisiting natural gradient for deep networks Abstract: The aim of this paper is three-fold. First we show that Hessian-Free (Martens, 2010) and Krylov Subspace Descent (Vinyals and Povey, 2012) can be described as implementations of natural gradient descent due to their use of the extended Gauss-Newton approximation of the Hessian. Secondly we re-derive natural gradient from basic principles, contrasting the difference between two versions of the algorithm found in the neural network literature, as well as highlighting a few differences between natural gradient and typical second order methods. Lastly we show empirically that natural gradient can be robust to overfitting and particularly it can be robust to the order in which the training data is presented to the model. paper on openreview<http://openreview.net/document/1cd7651c-8029-457e-ae24-5fbca0f3a6a7#1cd7651…> I hope to see many of you there, Best, Razvan

1 1

Tea talk Wednesday 2nd April @13:00 AA3195 by Chris Pal
by Razvan Pascanu 02 Avr '14

02 Avr '14

Hi all, This Wednesday, same place AA3195, same time 13:00, we have the following tea talk scheduled: Talk by : Christopher Pal Title: A New Smooth Approximation for Learning with the Zero One Loss with a Probabilistic Interpretation We examine a new form of smooth approximation to the zero one loss which can be viewed as learning using a reformulation of the classical functional form of logistic regression. Our approach is based on using the posterior mean of a novel generalized Beta-Bernoulli formulation. This leads to a generalized logistic function that approximates the zero one loss, but retains a probabilistic formulation. We show how the underlying functional form has a number of interesting properties. The approach is also easily generalized to kernel logistic regression and beyond. We present experiments in which we learn such models using a combination of gradient descent and coordinate descent using localized grid search so as to escape from local minima. Our experiments are indicating that that this formulation can improve upon the performance of canonical models such as classical logistic regression and support vector machines and is more robust to outliers. This work is with Md. Kamrul Hasan and he may present the second half of the talk. I hope to see many of you there, Best, Razvan

1 1

Tea talk Wednesday 26th @13:00 AA3195 by Mehdi
by Razvan Pascanu 24 Mar '14

24 Mar '14

Hi all, This Wednesday, 26th March, at 13:00 room AA3195 Mehdi Mirza is going to present (Ian's) and his submission to ICLR 2014 which was accepted in the conference track. See title and abstract bellow. Jessica Thompson will also present after Mehdi. She will introduce herself to the lab and talk about her work. I hope to see many of you there, Best, Razvan An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks Authors Ian J Goodfellow, Mehdi Mirza, Xia Da, Aaron Courville, Yoshua Bengio Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models "forget" how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting. We find that it is always best to train using the dropout algorithm--the dropout algorithm is consistently best at adapting to the new task, remembering the old task, and has the best tradeoff curve between these two extremes. We find that different tasks and relationships between tasks result in very different rankings of activation function performance. This suggests the choice of activation function should always be cross-validated.

1 0

Presentation this Friday 11:00 in AA3195 by Christian Gagné
by Razvan Pascanu 20 Mar '14

20 Mar '14

Hi all, This Friday, 11:00 we have a presentation by Christian Gagné from Universite Laval. The room is AA3195. I hope to see many of you there. Title: Evolutionary Computation and Machine Learning for the Engineering of Intelligent Systems Abstract: Evolutionary computation consists in black-box population-based optimization techniques inspired by natural evolution. These metaheuristics are quite successful at solving problems hard to tackle with conventional methods, for example when dealing with strongly non convex multimodal functions, multiple objectives, or solutions represented a complex data structures (e.g., graphs, trees, symbolic expressions). Although evolutionary computation is often considered as a close sibling to machine learning, it is often badly understood and dismissed by researchers in the field, mostly given its ad hoc nature and the impression it lacks strong theoretical foundations. Such considerations are unfortunate, as many criticisms are not taking into account latest developments of the field, nor considering the use of powerful tools provided by evolutionary computation for tackling some of the difficult optimization problems of machine learning. In my talk, I will present the field of evolutionary computation, with a particular emphasis on how it can be used in the context of machine learning. I will also present some methods of evolutionary computing that are relying on machine learning. This will be illustrated by various works done in my group at Université Laval, where we are applying in synergy machine learning and evolutionary computation techniques for solving complex engineering problems. Presenter: Christian Gagné Professeur agrégé Laboratoire de vision et systèmes numériques Département de génie électrique et de génie informatique Université Laval, Québec (Québec), Canada http://vision.gel.ulaval.ca/~cgagne christian.gagne(a)gel.ulaval.ca

1 1

Tea talk Wednesday 19th @13:00 AA3195 by Roland Memisevic
by Razvan Pascanu 17 Mar '14

17 Mar '14

Hi all, I'm glad to announce our next tea talk this Wednesday (19th March), at the usual place AA3195 at 13:00. Roland is going to talk to us about some of his recent work. See title and abstract below. I hope to see many of you there ! Title: Zero-bias autoencoders Abstract: We recently stumbled upon a hidden unit activation function that makes it very easy to train an autoencoder by simply minimizing reconstruction error, without using any additional regularization. The activation function takes the following form: it is zero if the net input to the neuron is smaller than some threshold and it is linear elsewhere. It seems important for the linear part to be truly linear and not affine, in other words the bias term for the linear part should be zero. I will describe some experiments as well as some theoretical insights that help explain why this activation function works so well. Joint work with Kishore Konda and David Krueger. In ASCII art: / / / / / / | | | --------0---------- hidden unit's net input on the x-axis, output on the y-axis. (this picture may be messed up if your mail-reader is smart about font spacing) Best, Razvan

1 0

Re: [Lisa_teatalk] [Lisa_labo] Who wants to do a tea talk?
by Ian Goodfellow 13 Mar '14

13 Mar '14

My 2nd and 3rd author are feeling left out after all of the popular press articles only mentioned me, so I'm going to let one (or both, if that's OK for the ICLR format) of them do the presentation. 2014-03-13 12:18 GMT-04:00 David Krueger <david.scott.krueger(a)gmail.com>: > Guillaume - Monty Hall problem: you say not to switch doors? Or it > doesn't matter? > > > On Thu, Mar 13, 2014 at 11:38 AM, Guillaume Alain <gyomalin(a)gmail.com>wrote: > >> >> If you're looking for someone to present something after Medhi, I can >> make a short presentation about something random that I find interesting. >> >> I've got some ideas : >> - entropy and Battleship (not the movie) >> - bloom filters (the answer to every possible job interview question >> when the answer is not "just use hash maps") >> - bitcoins crash course (what's cool about the adversarial consensus >> achieved by the computer algorithm) >> - Monty Hall problem (why I think that the commonly accepted solution >> is wrong (or useless)) >> >> >> On 2014-03-13, at 11:09, Razvan Pascanu <r.pascanu(a)gmail.com> wrote: >> >> Hi guys, >> >> First of all, those with accepted papers at ICLR. >> >> I know I and Ian have an oral. Does anybody else got his paper accepted >> as an oral. If so and you want to do a practice talk let me know to reserve >> a spot. Ian, do you think you need a practice talk? >> >> Otherwise, those with posters, if you think having a tea talk will help >> you talk about the poster, or if you want to present your work, let me know >> so that I can schedule you. >> >> For now we have: >> next week (19th) is me + Yann talking about non-convex optimizations >> and saddle points >> 26th we have Mehdi talking about his paper at ICLR (catastrophic >> forgetting). >> 2nd of April I scheduled myself to do a practice talk for my oral >> (natural gradient) >> >> Mehdi said his talk is going to be short, so he would not mind if someone >> else wants to slit that time slot with him. Anybody wants to do that ? >> >> If no one with ICLR paper wants to present their work, is there someone >> else who wants to present either: >> >> (a) an idea they are working on >> (b) some submitted work somewhere (or just tech report) >> (c) some very interesting paper they read recently >> (d) just introduce themselves to the lab talking of what they've done >> before coming here >> >> >> Thanks a lot, >> Best, >> Razvan >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >> >> >> >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo >> >> > > _______________________________________________ > Lisa_labo mailing list > Lisa_labo(a)iro.umontreal.ca > https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo > >

2 2

Who wants to do a tea talk?
by Razvan Pascanu 13 Mar '14

13 Mar '14

Hi guys, First of all, those with accepted papers at ICLR. I know I and Ian have an oral. Does anybody else got his paper accepted as an oral. If so and you want to do a practice talk let me know to reserve a spot. Ian, do you think you need a practice talk? Otherwise, those with posters, if you think having a tea talk will help you talk about the poster, or if you want to present your work, let me know so that I can schedule you. For now we have: next week (19th) is me + Yann talking about non-convex optimizations and saddle points 26th we have Mehdi talking about his paper at ICLR (catastrophic forgetting). 2nd of April I scheduled myself to do a practice talk for my oral (natural gradient) Mehdi said his talk is going to be short, so he would not mind if someone else wants to slit that time slot with him. Anybody wants to do that ? If no one with ICLR paper wants to present their work, is there someone else who wants to present either: (a) an idea they are working on (b) some submitted work somewhere (or just tech report) (c) some very interesting paper they read recently (d) just introduce themselves to the lab talking of what they've done before coming here Thanks a lot, Best, Razvan

3 3

Tea talk Wednesday (12th March) -- 13:00 AA 3195
by Razvan Pascanu 12 Mar '14

12 Mar '14

Hi all, This Wednesday 13:00 room AA 3195, I will give a tea talk on my work with Guido Montufar, KyungHyun Cho and Yoshua regarding the importance of depth for feedforward neural networks with piece-wise linear activation functions. This work (split into two papers) was submitted to ICLR and COLT this year. Ttile: On the number of linear regions of deep neural networks Abstract: We study the complexity of functions computable by deep feedforward neural networks with piece-wise linear activations in terms of the number of regions of linearity that they have. Deep networks are able to sequentially map portions of each layer's input space to the same output. In this way, deep models compute functions with a compositional structure that is able to re-use pieces of computation exponentially often in terms of their depth. This note investigates the complexity of such compositional maps and contributes new theoretical results regarding the advantage of depth for neural networks with piece-wise linear activation functions. paper submitted to COLT : http://arxiv.org/abs/1402.1869 paper submitted to ICLR : http://arxiv.org/abs/1312.6098 Hope to see many of you there, Razvan

1 1

Tea talk tomorrow 13:00 (changed room to AA3256)
by Razvan Pascanu 25 Fév '14

25 Fév '14

Hi all, This week Ian is going to talk about his ICLR submission. The room is going to be the main Lisa Lab (AA 3256). Hope to see many of you there. Title: Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks Abstract: A tea talk on this ICLR submission: https://drive.google.com/file/d/0B64011x02sIkd3RwSDRpTXlKSzQ/edit?usp=shari… Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a uniﬁed approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief (Dean et al., 2012) implementation of deep neural networks in order to train large, distributed neural networks on high quality images. We ﬁnd that the performance of this approach increases with the depth of the convolutional network, with the best performance occurring in the deepest architecture we trained, with eleven hidden layers. We evaluate this approach on the publicly available SVHN dataset and achieve over 96% accuracy in recognizing complete street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art and achieve 97.84% accuracy. We also evaluate this approach on an even more challenging dataset generated from Street View imagery containing several tens of millions of street number annotations and achieve over 90% accuracy. Our evaluations further indicate that at speciﬁc operating thresholds, the performance of the proposed system is comparable to that of human operators. To date, our system has helped us extract close to 100 million physical street numbers from Street View imagery worldwide. Best, Razvan

1 1

Presentation this Friday 11:00 in AA3195
by Razvan Pascanu 21 Fév '14

21 Fév '14

Hi all, This Friday, 10:00 we have a presentation by Christian Gagné from Universite Laval. The room is AA3195. I hope to see many of you there. Title: Evolutionary Computation and Machine Learning for the Engineering of Intelligent Systems Abstract: Evolutionary computation consists in black-box population-based optimization techniques inspired by natural evolution. These metaheuristics are quite successful at solving problems hard to tackle with conventional methods, for example when dealing with strongly non convex multimodal functions, multiple objectives, or solutions represented a complex data structures (e.g., graphs, trees, symbolic expressions). Although evolutionary computation is often considered as a close sibling to machine learning, it is often badly understood and dismissed by researchers in the field, mostly given its ad hoc nature and the impression it lacks strong theoretical foundations. Such considerations are unfortunate, as many criticisms are not taking into account latest developments of the field, nor considering the use of powerful tools provided by evolutionary computation for tackling some of the difficult optimization problems of machine learning. In my talk, I will present the field of evolutionary computation, with a particular emphasis on how it can be used in the context of machine learning. I will also present some methods of evolutionary computing that are relying on machine learning. This will be illustrated by various works done in my group at Université Laval, where we are applying in synergy machine learning and evolutionary computation techniques for solving complex engineering problems. Presenter: Christian Gagné Professeur agrégé Laboratoire de vision et systèmes numériques Département de génie électrique et de génie informatique Université Laval, Québec (Québec), Canada http://vision.gel.ulaval.ca/~cgagne christian.gagne(a)gel.ulaval.ca

3 4

Tea talk Wednesday 13:00 AA 3195
by Razvan Pascanu 18 Fév '14

18 Fév '14

Hi all, This Wednesday, at 13:00, room AA 3195 (Andre-Aisenstadt) Guillaume Desjardins is going to talk about two interesting papers coming from DeepMind. (1) Andriy Mnih's Neural Variational Inference ( http://arxiv.org/abs/1402.0030) and (2) Karol's Deep AutoRegressive Networks (http://arxiv.org/abs/1310.8499) Hope to see you there. Razvan

1 0

Thesis defence - Guillaume Desjardins Thursday 15:30 AA3195
by Razvan Pascanu 18 Fév '14

18 Fév '14

Hi all, It gives me great pleasure to announce Guillaume's defence. It is this Thursday 15:30 in AA3195. Here is the title and abstract: Improving Sampling, Optimization and Feature Extraction in Boltzmann Machines Despite the current widescale success of deep learning in training large scale hierarchical models through supervised learning, unsupervised learning promises to play a crucial role towards solving general Artificial Intelligence, where agents are expected to learn with little to no supervision. The work presented in this thesis tackles the problem of unsupervised feature learning and density estimation, using a model family at the heart of the deep learning phenomenon: the Boltzmann Machine (BM). We present contributions in the areas of sampling, partition function estimation, optimization and the more general topic of invariant feature learning. With regards to sampling, we present a novel adaptive parallel tempering method which dynamically adjusts the temperatures under simulation to maintain good mixing in the presence of complex multi-modal distributions. When used in the context of (stochastic) maximum likelihood (SML) training, the improved ergodicity of our sampler translates to increased robustness to learning rates and faster per epoch convergence. Though our application is limited to BM, our method is general and is applicable to sampling from arbitrary probabilistic models using Markov Chain Monte Carlo (MCMC) techniques. While SML gradients can be estimated via sampling, computing data likelihoods requires an estimate of the partition function. Contrary to previous approaches which consider the model as a black box, we provide an efficient algorithm which instead tracks the change in log partition function incurred by successive parameter updates. Our algorithm frames this estimation problem as one of filtering performed over a 2D lattice, with one dimension representing time and the other temperature. On the topic of optimization, our thesis presents a novel algorithm for applying the natural gradient to large scale Boltzmann Machines. Up until now, its application had been constrained by the computational and memory requirements of computing the Fisher Information Matrix (FIM), which is square in the number of parameters. The Metric-Free Natural Gradient algorithm (MFNG) avoids computing the FIM altogether by combining a linear solver with an efficient matrix-vector operation. The method shows promise in that the resulting updates yield faster per-epoch convergence, despite being slower in terms of wall-time. Finally, we explore how invariant features can be learnt through modifications to the BM energy function. We study the problem in the context of the spike \& slab RBM, which we extend to handle both binary and sparse input distributions. By associating each spike with several slab variables, latent variables can be made invariant to a rich, high dimensional subspace resulting in increased invariance in the learnt representation. When using the expected model posterior as input to a classifier, increased invariance translates to improved classification accuracy in the low-label data regime. We conclude by showing a connection between invariance and the more powerful concept of disentangling factors of variation. While invariance can be achieved by pooling over subspaces, disentangling can be achieved by learning multiple complementary views of the same subspace. In particular, we show how this can be achieved using third-order BMs featuring multiplicative interactions between pairs of random variables. Hope to see many of you there ! Razvan

1 1

Re: [Lisa_teatalk] Tea talk Thu 13th Feb, 14h-15h in Z300
by Razvan Pascanu 11 Fév '14

11 Fév '14

Sorry for spamming but the title of the email might have been a bit confusing. To emphasize: Tea talk is **Thursday 13th Feb**. Razvan On Tue, Feb 11, 2014 at 8:28 AM, Razvan Pascanu <r.pascanu(a)gmail.com> wrote: > Hi all, > > Thursday we have a tea talk by Jyri Kivinen. The room is going to be Z300 > McNicoll pavilion, time 14h00 to 15h00. PLEASE NOTE THE CHANGE OF TIME AND > PLACE. > > Here is the title and abstract: > > Title: > Statistical Models for Natural Scene Data > > Abstract: > This talk describes work largely described in my PhD thesis (University of > Edinburgh, 2013) titled Statistical Models for Natural Scene Data. > > We consider statistical modelling of natural image data. Obtaining > advances in this field can have significant impact for both engineering > applications, and for the understanding of the human visual system. Several > recent advances in natural image modeling have been obtained with the use > of unsupervised feature learning. We consider a class of such models, > restricted Boltzmann machines (RBMs), used in many recent state-of-the-art > image models. We develop extensions of these stochastic artificial neural > networks, and use them as a basis for building more effective image models, > and tools for computational vision. > > First, motivated by the unsatisfactory results of current generative > natural image models, we take a step back, and evaluate whether they are > able to model a subclass of the data, natural image textures. This is a > necessary subcomponent of any credible model for visual scenes. We assess > the performance of a state-of-the-art model of natural images for texture > generation, using a dataset and evaluation techniques from in prior work. > We also perform a dissection of the model architecture, uncovering the > properties important for good performance. Building on this, we develop > structured extensions for more complicated data comprised of textures from > multiple classes, using the single-texture > model architecture as a basis. These models are shown to be able to > produce state-of-the-art texture synthesis results quantitatively, and are > also effective qualitatively. It is demonstrated empirically that the > developed multiple-texture framework provides a means to generate images of > differently textured regions, more generic globally varying textures, and > can also be used for texture interpolation, where the approach is radically > different from the others in the area. > > We then consider visual boundary prediction from natural images. The work > aims to improve understanding of Boltzmann machines in the generation of > image segment boundaries, and to investigate deep neural network > architectures for learning the boundary detection problem. The developed > networks (which avoid several hand-crafted model and feature designs > commonly used for the problem), produce effective inference with > state-of-the-art or better performance. > > Finally we discuss a novel framework we have developed for obtaining > Boltzmann machines, in which the hidden unit activations co-transform with > transformed input stimuli in a stable and predictable way throughout the > network. We define such models to be \emph{transformation equivariant}. > Such properties have been shown useful for computer vision systems, and > have been motivational for example in the development of steerable filters, > a widely used classical feature extraction technique. Translation > equivariant feature sharing has been the standard method for scaling image > models beyond patch-sized data to large images. In our framework we extend > shallow and deep models to account for other kinds of transformations as > well, focusing on in-plane rotations. > > I hope to see many of you there ! > > Best, > Razvan >

1 0

Tea talk tomorrow Thu 13th Feb, 14h-15h in Z300
by Razvan Pascanu 11 Fév '14

11 Fév '14

Hi all, Thursday we have a tea talk by Jyri Kivinen. The room is going to be Z300 McNicoll pavilion, time 14h00 to 15h00. PLEASE NOTE THE CHANGE OF TIME AND PLACE. Here is the title and abstract: Title: Statistical Models for Natural Scene Data Abstract: This talk describes work largely described in my PhD thesis (University of Edinburgh, 2013) titled Statistical Models for Natural Scene Data. We consider statistical modelling of natural image data. Obtaining advances in this field can have significant impact for both engineering applications, and for the understanding of the human visual system. Several recent advances in natural image modeling have been obtained with the use of unsupervised feature learning. We consider a class of such models, restricted Boltzmann machines (RBMs), used in many recent state-of-the-art image models. We develop extensions of these stochastic artificial neural networks, and use them as a basis for building more effective image models, and tools for computational vision. First, motivated by the unsatisfactory results of current generative natural image models, we take a step back, and evaluate whether they are able to model a subclass of the data, natural image textures. This is a necessary subcomponent of any credible model for visual scenes. We assess the performance of a state-of-the-art model of natural images for texture generation, using a dataset and evaluation techniques from in prior work. We also perform a dissection of the model architecture, uncovering the properties important for good performance. Building on this, we develop structured extensions for more complicated data comprised of textures from multiple classes, using the single-texture model architecture as a basis. These models are shown to be able to produce state-of-the-art texture synthesis results quantitatively, and are also effective qualitatively. It is demonstrated empirically that the developed multiple-texture framework provides a means to generate images of differently textured regions, more generic globally varying textures, and can also be used for texture interpolation, where the approach is radically different from the others in the area. We then consider visual boundary prediction from natural images. The work aims to improve understanding of Boltzmann machines in the generation of image segment boundaries, and to investigate deep neural network architectures for learning the boundary detection problem. The developed networks (which avoid several hand-crafted model and feature designs commonly used for the problem), produce effective inference with state-of-the-art or better performance. Finally we discuss a novel framework we have developed for obtaining Boltzmann machines, in which the hidden unit activations co-transform with transformed input stimuli in a stable and predictable way throughout the network. We define such models to be \emph{transformation equivariant}. Such properties have been shown useful for computer vision systems, and have been motivational for example in the development of steerable filters, a widely used classical feature extraction technique. Translation equivariant feature sharing has been the standard method for scaling image models beyond patch-sized data to large images. In our framework we extend shallow and deep models to account for other kinds of transformations as well, focusing on in-plane rotations. I hope to see many of you there ! Best, Razvan

1 0

Tea talk tomorrow Wed 5th Feb, 13h-14h in AA3195
by Razvan Pascanu 04 Fév '14

04 Fév '14

Hi all, Tomorrow we have a tea talk by Philemon Brakel. The room is going to be Andre Aisenstadt room 3195, time 13h00 to 14h00. Here is the title and abstract: Multi-tempering and training energy models for missing value imputation. First I will describe an extension of Parallel Tempering Markov chain Monte Carlo and show how it allows the method to make more efficient use of the information the sampling chains provide. Second, I will talk about some work I did on training temporal energy models for imputing missing values. Finally, I would like to discuss some preliminary ideas for learning intractable generative models. I hope to see many of you there ! Best, Razvan

1 0

Tea talk tomorrow Wednesday 22, from 13-14
by Razvan Pascanu 22 Jan '14

22 Jan '14

Hi all, Here is the announcement for the tea talk tomorrow: Speaker: Yann and Razvan Title: More research ideas Abstract: In the same spirit as Ian previous tea talk, we will go over a few research ideas. They will range from optimization ideas (extension of standard second order methods), sampling from GSNs, recurrent neural networks tweaks and simple ideas of scaling things up. I hope to see many of you there. Razvan

2 2