- Lisa_teatalk - mailman.iro.umontreal.ca

Tea talk tomorrow Wednesday 22, from 13-14
by Razvan Pascanu 22 Jan '14

22 Jan '14

Hi all, Here is the announcement for the tea talk tomorrow: Speaker: Yann and Razvan Title: More research ideas Abstract: In the same spirit as Ian previous tea talk, we will go over a few research ideas. They will range from optimization ideas (extension of standard second order methods), sampling from GSNs, recurrent neural networks tweaks and simple ideas of scaling things up. I hope to see many of you there. Razvan

2 2

talk this Thursday 16th 13:00 - Ian Goodfellow
by Razvan Pascanu 16 Jan '14

16 Jan '14

Hi all, First of all the schedule has changed, and tea talks will be Wednesdays from 13-14 from now on. There is an exception this week. We will have Ian presenting this **Thursday from 13:00 to 14:00**. Speaker: Ian Goodfellow Title: Ian's ideas for research projects Abstract: I'll throw out a few of my recent ideas for research projects, ranging from easy to idealistic: -Restricted maxout units -Regularizing piecewise linear nets to change pieces rarely -Sparsely connected recurrent nets -Trajectory optimization for recurrent nets -"Cognitive agency" and how you can use it to do things like discrete-state nets, dynamically structured nets, and nets that interact with data structures like stacks and tapes I hope to see many of you there. And please register for giving tea talks. Razvan

4 3

Talk by Surya Ganguli, this Friday 17th Jan, 14h00-15h00 AA3195
by Razvan Pascanu 14 Jan '14

14 Jan '14

Hi all, This Friday we have a talk by Surya Ganguli at 14h00 in room 3195 Andre Aisenstadt. Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Networks Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed remains finite: for a special class of initial conditions on the weights, very deep networks incur only a finite delay in learning speed relative to shallow networks. We further show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions. We also discuss application to infant semantic development. I hope to see many of you there ! Best, Razvan

2 1

Practice talk, Combining Modality Specific Deep Nets for Emotion Recognition in Video
by Razvan Pascanu 30 Nov '13

30 Nov '13

This is a practice talk that Samira will be giving at ICMI in Sydney, presenting the results of our team on the EmotiW challenge. It's a short presentation (20 minutes), and we would like you to be present and give her feedback. Title: Combining Modality Speciﬁc Deep Neural Networks for Emotion Recognition in Video Speaker: Samira Ebrahimi Kahou Location: AA-1409 Time: Tuesday, December 3rd, 3 p.m. Abstract of the article: In this paper we present the techniques used for the University of Montreal's team submissions to the 2013 Emotion Recognition in the Wild Challenge. The challenge is to classify the emotions expressed by the primary human subject in short video clips extracted from feature length movies. This involves the analysis of video clips of acted scenes lasting approximately one-two seconds, including the audio track which may contain human voices as well as background music. Our approach combines multiple deep neural networks for different data modalities, including: (1) a deep convolutional neural network for the analysis of facial expressions within video frames; (2) a deep belief net to capture audio information; (3) a deep autoencoder to model the spatio-temporal information produced by the human actions depicted within the entire scene; and (4) a shallow network architecture focused on extracted features of the mouth of the primary human subject in the scene. We discuss each of these techniques, their performance characteristics and different strategies to aggregate their predictions. Our best single model was a convolutional neural network trained to predict emotions from static frames using two large data sets, the Toronto Face Database and our own set of faces images harvested from Google image search, followed by a per frame aggregation strategy that used the challenge training data. This yielded a test set accuracy of 35.58%. Using our best strategy for aggregating our top performing models into a single predictor we were able to produce an accuracy of 41.03% on the challenge test set. These compare favorably to the challenge baseline test set accuracy of 27.56%. Hope to see many of you there, Razvan

1 0

Tea talk this Friday 3PM
by Razvan Pascanu 20 Nov '13

20 Nov '13

Hi all, This week we have a tea talk where Caglar, Kyunghyun and myself will present the work submitted at AISTATS : Lp units for MLPs. I originally wanted to do this last week, but had to reschedule for this week. Also note that Friday 3PM will be the new tea-talk time (at least for now). The talk will try to go beyond what with submitted. We will try to summarize some ideas we have to extend this work, namely my allowing more flexibility in the units. I will also talk about how one can visualize what these units are doing and how that could be useful. The abstract of the paper: In this paper we proposed a novel nonlinear unit, which is called as Lp unit, for a multi-layer perceptron (MLP). The proposed Lp unit receives signal from several projections of the layer below and computes the normalized Lp norm. We notice two interesting interpretations of the Lp unit. First, we note that the proposed unit is a generalization of a number of conventional pooling operators such as average, root-mean-square and max pooling widely used in, for instance, convolutional neural networks(CNN), HMAX models and neocognitrons. Furthermore, under certain constraints, the Lp unit is a generalization of the recently proposed maxout unit(Goodfellow et al, 2013) which achieved the state-of-the-art object recognition results on a number of benchmark datasets. Second, we provide a geometrical interpretation of the activation function. Each Lp unit defines a spherical boundary, with its exact shape defined by the order p. We claim that this makes it possible to obtain arbitrarily shaped, curved boundaries more efficiently by combining just a few Lp units of different orders. We empirically evaluate the proposed Lp units on a number of datasets and show that MLPs consisting of the Lp units achieves the state-of-the-art results on a number of benchmark datasets. Hope to see many of you there, Razvan

1 0

Talk by Hugo Tuesday 19th in Z300 at 10:30
by Razvan Pascanu 18 Nov '13

18 Nov '13

Hi all, This Tuesday, 19th November, Hugo Larochelle is visiting our lab and is going to give a talk. The time is 10:30 AM, place is Z300, pavilion McNicoll. Title: Deep Learning for Distribution Estimation Abstract:: In this talk, I will present my recent research on using neural networks for the task of distribution/density estimation, a fundamental problem in machine learning. Specifically, I will discuss the neural autoregressive distribution estimator (NADE), a state-of-the-art estimator of the probability distribution of data along with its various extensions. I will also describe a recent deep version of NADE, which again illustrates the statistical modelling power of deep models. Hope to see many of you there, Best, Razvan

2 2

Tea talk this week Friday 3PM. room AA 3195
by Razvan Pascanu 14 Nov '13

14 Nov '13

The room has been confirmed. Is AA 3195. Best, Razvan On Wed, Nov 13, 2013 at 10:46 PM, Razvan Pascanu <r.pascanu(a)gmail.com>wrote: > > > Hi everyone, > > We've change the speaker for this tea talk. Hal is going to give a talk > instead. The room remains TBD, but same time 15:00 Friday. > > Hope to see many of you there. > Thanks. > > > title: A picture is worth 13.6 words (on average) > > abstract: I'll discuss some recent work I've been involved in that ties > language processing (which I know something about) and computer vision > (which I do not). Technically I'll focus on problems related to caption > generation. I'll highlight some of the challenges, discuss some successes > (and more failures) and discuss what I think the interesting problems are > moving forward. This talk is very informal, and feedback his highly > appreciated! This will include ideas and results that evolved from joint > work with Yiannis Aloimonos, Alex Berg, Tamara Berg, Jesse Dodge, Aleks > Ecins, Cornelia Fermuller, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret > Mitchell, Karl Stratos, Ching Lik Teo, Kota Yamaguchi and Yezhou Yang. > > >

1 0

Tea talk this week Friday 3PM. room TBD [note change of speaker]
by Razvan Pascanu 14 Nov '13

14 Nov '13

Hi everyone, We've change the speaker for this tea talk. Hal is going to give a talk instead. The room remains TBD, but same time 15:00 Friday. Hope to see many of you there. Thanks. title: A picture is worth 13.6 words (on average) abstract: I'll discuss some recent work I've been involved in that ties language processing (which I know something about) and computer vision (which I do not). Technically I'll focus on problems related to caption generation. I'll highlight some of the challenges, discuss some successes (and more failures) and discuss what I think the interesting problems are moving forward. This talk is very informal, and feedback his highly appreciated! This will include ideas and results that evolved from joint work with Yiannis Aloimonos, Alex Berg, Tamara Berg, Jesse Dodge, Aleks Ecins, Cornelia Fermuller, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Karl Stratos, Ching Lik Teo, Kota Yamaguchi and Yezhou Yang. On Wed, Nov 13, 2013 at 9:18 AM, Razvan Pascanu <r.pascanu(a)gmail.com> wrote: > Hi all, > > This week we will have a tea talk where Caglar, Kyunghyun and myself will > present the work submitted at AISTATS : Lp units for MLPs. > > The talk will try to go beyond what with submitted. We will try to > summarize some ideas we have to extend this work, namely my allowing more > flexibility in the units. I will also talk about how one can visualize what > these units are doing and how that could be useful. > > The abstract of the paper: > > In this paper we proposed a novel nonlinear unit, which is called as Lp > unit, for a multi-layer perceptron (MLP). The proposed Lp unit receives > signal from several projections of the layer below and computes the > normalized Lp norm. We notice two interesting interpretations of the Lp > unit. First, we note that the proposed unit is a generalization of a number > of conventional pooling operators such as average, root-mean-square and max > pooling widely used in, for instance, convolutional neural networks(CNN), > HMAX models and neocognitrons. Furthermore, under certain constraints, the > Lp unit is a generalization of the recently proposed maxout unit > (Goodfellow et al, 2013) which achieved the state-of-the-art object > recognition results on a number of benchmark datasets. Second, we provide a > geometrical interpretation of the activation function. Each Lp unit defines > a spherical boundary, with its exact shape defined by the order p. We claim > that this makes it possible to obtain arbitrarily shaped, curved boundaries > more efficiently by combining just a few Lp units of different orders. We > empirically evaluate the proposed Lp units on a number of datasets and show > that MLPs consisting of the Lp units achieves the state-of-the-art results > on a number of benchmark datasets. > > > Hope to see many of you there, > > Razvan >

1 0

Tea talk this Week Friday 3PM in LISA room
by Razvan Pascanu 13 Nov '13

13 Nov '13

Hi all, This week we will have a tea talk where Caglar, Kyunghyun and myself will present the work submitted at AISTATS : Lp units for MLPs. The talk will try to go beyond what with submitted. We will try to summarize some ideas we have to extend this work, namely my allowing more flexibility in the units. I will also talk about how one can visualize what these units are doing and how that could be useful. The abstract of the paper: In this paper we proposed a novel nonlinear unit, which is called as Lp unit, for a multi-layer perceptron (MLP). The proposed Lp unit receives signal from several projections of the layer below and computes the normalized Lp norm. We notice two interesting interpretations of the Lp unit. First, we note that the proposed unit is a generalization of a number of conventional pooling operators such as average, root-mean-square and max pooling widely used in, for instance, convolutional neural networks(CNN), HMAX models and neocognitrons. Furthermore, under certain constraints, the Lp unit is a generalization of the recently proposed maxout unit (Goodfellow et al, 2013) which achieved the state-of-the-art object recognition results on a number of benchmark datasets. Second, we provide a geometrical interpretation of the activation function. Each Lp unit defines a spherical boundary, with its exact shape defined by the order p. We claim that this makes it possible to obtain arbitrarily shaped, curved boundaries more efficiently by combining just a few Lp units of different orders. We empirically evaluate the proposed Lp units on a number of datasets and show that MLPs consisting of the Lp units achieves the state-of-the-art results on a number of benchmark datasets. Hope to see many of you there, Razvan

1 0

**CHANGE OF TIME TO THURSDAY 31 Oct at 13:00** room Z-260 Tea talk
by Razvan Pascanu 29 Oct '13

29 Oct '13

Hi guys, There has been a change of date/room to Thursday 31 Oct 13:00 room Z-260 as Yoshua is leaving Friday morning to Korea. Razvan On Sat, Oct 26, 2013 at 12:12 AM, Razvan Pascanu <r.pascanu(a)gmail.com>wrote: > > Next > week > we will have Xiaohua Liu giving a tea talk. NOTE the change of day to > Friday and time 15:00. Please find below the title and abstract of the > talk, as well as short introduction of Xiaohua. > Hope to see many of you there ! > > > > Title: > > Collective Information Extraction for Micro-blogs**** > > ** ** > > Abstract:**** > > > Within the past five years, micro-blogs have greatly developed and now > become one indispensable information source. Owing to the unstructured, > fragmented, informally written, noisy and real-time nature of micro-blogs, > it is hard for people to effectively obtain the information of their > interests from micro-blogs. **** > > ** ** > > In this talk, I will introduce the task of information extraction for > micro-blogs, which consists of a serials of sub-tasks, including Named > Entity Recognition (NER), Semantic Role Labeling (SRL) and Event Extraction > (EE). **** > > ** ** > > One main common research challenge is the dearth of information in a > single micro-blog, rooted in the short and noisy nature of micro-blogs. In > this talk, I will introduce the following three kinds of collective > inference algorithms, which are applied to NER, SRL and EE, respectively, > achieving remarkable improvements over the baselines.**** > > ** ** > > 1. K-nearest neighbor (KNN) classifier based joint inference that > determines the output of the current input using the outputs of the inputs > similar to the current input;**** > > 2. Cluster based joint inference that involves the following two-stage > labeling: In the first round, one single micro-blog is processed as usual; > in the second round, similar micro-blogs are first clustered, then for each > cluster an enhanced model that utilizes cluster-level information is > applied to each micro-blog in the cluster;**** > > 3. Graphical model based joint inference that conducts an information > extraction related task on a collection of related micro-blogs, sharing > information across micro-blogs and simultaneously determining the outputs > for all the related micro-blogs.**** > > ** ** > > In the end, I will present a semantic search of micro-blogs called > QuickView, which integrates various information extraction technologies and > enables categorized browsing and advanced search of micro-blogs.**** > > ** ** > > A few words about myself:**** > > ** ** > > I am an affiliated postdoctoral research fellow of Professor Jian-Yun’s > team. Before I moved to Montreal, I was Project Lead and Researcher in > Microsoft Research Asia, Natural Language Computing Group. My research > interests include big data processing, social content mining, NLP for > ranking, machine learning, etc. In recent three years I published 30+ > papers in ACL, AAAI, IJCAI, EMNLP, COLING, and SIGIR. At my free time, I > like coding and solving software development related challenges. I authored > several programming related books, including "Detailed introduction of JSP > programming" and "Mastering MFC". I received PhD from Harbin Institute > Technology, in 2012.**** > > ** > > >

1 0

Talks by Chris Williams Monday 28th AA 3195
by Razvan Pascanu 28 Oct '13

28 Oct '13

Hi all, Monday 28th October we will have Chris Williams giving to talks. Both of them will be in the rmm AA3195 (pavillon Andre-Aisendstandt, click here<http://maps.google.ca/maps?q=2920,+chemin+de+la+tour&hl=en&z=16&iwloc=A> for directions) at the Université de Montréal. Abstract and title below. Hope to see you there ! First talk at 10:30 in AA3195 Title: Missing Components in Visual Scene Understanding: Hierarchy and Perceptual Organization? Chris Williams, School of Informatics, University of Edinburgh There has been considerable progress in object recognition and scene understanding over the last decade. However, we argue that much of this work uses relatively shallow architectures, and that more attention needs to be paid to hierarchical models and to processes of perceptual organization, within a probabilistic inference framework. ----------------------- Second talk at 13:30 in AA3195 Title: Switching Linear Dynamical Systems for Condition Monitoring in the Intensive Care Unit Abstract: Data drawn from an observed system is often usefully described by a number of hidden (or latent) factors. Given a sequence of observations, the task is to infer which latent factors are active at each time frame. In this talk I will describe the application of a switching linear dynamical model to monitoring the condition of a patient receiving intensive care. The state of health of a patient cannot be observed directly, but different underlying factors are associated with particular patterns of measurements, e.g. in the heart rate, blood pressure and temperature. We demonstrate how to exploit knowledge of the structure of how the various latent factors interact so as to reduce the amount of training data needed for the system. A combination of domain knowledge engineering and learning is used to produce an effective solution. We use the model to infer the presence of two different types of factors: common, recognisable regimes (e.g. certain artifacts or common physiological phenomena), and novel patterns which are clinically significant but have unknown cause. Experimental results are given showing the potential of the system for the early detection of neonatal sepsis, a major clinical concern in the care of premature babies. Chris Williams. Joint work with Yvonne Freer, Neil McIntosh, John Quinn, Ioan Stanculescu.

2 3

Tea talk next Friday 1st november 15:00
by Razvan Pascanu 26 Oct '13

26 Oct '13

Next week we will have Xiaohua Liu giving a tea talk. NOTE the change of day to Friday and time 15:00. Please find below the title and abstract of the talk, as well as short introduction of Xiaohua. Hope to see many of you there ! Title: Collective Information Extraction for Micro-blogs**** ** ** Abstract:**** Within the past five years, micro-blogs have greatly developed and now become one indispensable information source. Owing to the unstructured, fragmented, informally written, noisy and real-time nature of micro-blogs, it is hard for people to effectively obtain the information of their interests from micro-blogs. **** ** ** In this talk, I will introduce the task of information extraction for micro-blogs, which consists of a serials of sub-tasks, including Named Entity Recognition (NER), Semantic Role Labeling (SRL) and Event Extraction (EE). **** ** ** One main common research challenge is the dearth of information in a single micro-blog, rooted in the short and noisy nature of micro-blogs. In this talk, I will introduce the following three kinds of collective inference algorithms, which are applied to NER, SRL and EE, respectively, achieving remarkable improvements over the baselines.**** ** ** 1. K-nearest neighbor (KNN) classifier based joint inference that determines the output of the current input using the outputs of the inputs similar to the current input;**** 2. Cluster based joint inference that involves the following two-stage labeling: In the first round, one single micro-blog is processed as usual; in the second round, similar micro-blogs are first clustered, then for each cluster an enhanced model that utilizes cluster-level information is applied to each micro-blog in the cluster;**** 3. Graphical model based joint inference that conducts an information extraction related task on a collection of related micro-blogs, sharing information across micro-blogs and simultaneously determining the outputs for all the related micro-blogs.**** ** ** In the end, I will present a semantic search of micro-blogs called QuickView, which integrates various information extraction technologies and enables categorized browsing and advanced search of micro-blogs.**** ** ** A few words about myself:**** ** ** I am an affiliated postdoctoral research fellow of Professor Jian-Yun’s team. Before I moved to Montreal, I was Project Lead and Researcher in Microsoft Research Asia, Natural Language Computing Group. My research interests include big data processing, social content mining, NLP for ranking, machine learning, etc. In recent three years I published 30+ papers in ACL, AAAI, IJCAI, EMNLP, COLING, and SIGIR. At my free time, I like coding and solving software development related challenges. I authored several programming related books, including "Detailed introduction of JSP programming" and "Mastering MFC". I received PhD from Harbin Institute Technology, in 2012.**** **

2 1

Re: [Lisa_teatalk] Tea talk this Thursday 17th October CHANGE OF ROOM Z-315 13:00-15:00
by Razvan Pascanu 17 Oct '13

17 Oct '13

Sorry for the late notice. We only have the original room for an hour, so I've reserved a different one for two hours. Room Z-315 McNicoll from 13:00 to 15:00. Razvan On Wed, Oct 16, 2013 at 11:19 AM, Razvan Pascanu <r.pascanu(a)gmail.com>wrote: > This week Yoshua is going to give a talk. The room will be McNicoll Z-215. > Hope to see many of you there. > > > Title: > Deep learning, manifold learning and the challenge of normalization > constants > > Abstract: > > I will present some of the material for the four talks that I will give > next week at the UCL masterclass series and talk at Oxford, U.K. This > will include general material about deep learning, a description of the > geometrical view on unsupervised learning shared by manifold learning > algorithms and regularized auto-encoders, a discussion of the challenges > involved in general-purpose modeling of probability distributions due the a > large number of highly separated modes, and how depth and the framework of > generative stochastic networks can help. > > > Best, > Razvan >

1 0

Tea talk this Thursday 17th October
by Razvan Pascanu 16 Oct '13

16 Oct '13

This week Yoshua is going to give a talk. The room will be McNicoll Z-215. Hope to see many of you there. Title: Deep learning, manifold learning and the challenge of normalization constants Abstract: I will present some of the material for the four talks that I will give next week at the UCL masterclass series and talk at Oxford, U.K. This will include general material about deep learning, a description of the geometrical view on unsupervised learning shared by manifold learning algorithms and regularized auto-encoders, a discussion of the challenges involved in general-purpose modeling of probability distributions due the a large number of highly separated modes, and how depth and the framework of generative stochastic networks can help. Best, Razvan

2 2

Tea talk this thursday 26th
by Razvan Pascanu 23 Sep '13

23 Sep '13

Hi all, This week tea talk is going to be given by Ouais Alsharif : Title:End-to-End Text Recognition with Hybrid HMM Maxout Models Abstract: The problem of detecting and recognizing text in natural scenes has proved to be more challenging than its counterpart in documents, with most of the previous work focusing on a single part of the problem. In this work, we propose new solutions to the character and word detection problems and then we show how to combine these solutions in an end-to-end text-recognition system. We do so by leveraging the recently created maxout networks along with hybrid HMM models that have proven useful for voice recognition. Using these elements, we build a fast, tunable and highly accurate recognition system that beats state of the art results on all the sub-problems for both the ICDAR 2003 and SVT benchmark datasets. Hope to see many of you there :) Best, Razvan

1 0

Re: [Lisa_teatalk] [Lisa_labo] Tea-talk: Thursday, September 19 @ 1pm
by Cho KyungHyun 19 Sep '13

19 Sep '13

In the case anyone is interested, I'm attaching the set of slides I used today at the teatalk. Cho ________________________________ From: lisa_labo-bounces(a)iro.umontreal.ca [lisa_labo-bounces(a)iro.umontreal.ca] on behalf of Razvan Pascanu [r.pascanu(a)gmail.com] Sent: Tuesday, September 17, 2013 9:41 PM To: Lisa Labo; lisa_teatalk(a)iro.umontreal.ca Subject: [Lisa_labo] Tea-talk: Thursday, September 19 @ 1pm Please join us for tomorrow's tea-talk by Cho. The room will be, as usual, the lisa lab. Speaker: Cho KyungHyun Title: Neural Networks in Helsinki and Why I like RBMs Abstract: I will start the talk by briefly presenting research activities related to machine learning and neural networks going on in Helsinki, especially that of the department of information and computer science at the Aalto University. From there on I will explain my interest toward restricted Boltzmann machines and other unsupervised neural networks, which is based on the observation that neural networks may be viewed as an excellent machinery for approximating hierarchical probabilistic inference. In due course, I will introduce two novel approaches for training Boltzmann machines. At the end my current research direction focusing on recurrent neural networks and their potential application to machine translation will be discussed. Razvan

1 0

Tea-talk: Thursday, September 19 @ 1pm
by Razvan Pascanu 17 Sep '13

17 Sep '13

Please join us for tomorrow's tea-talk by Cho. The room will be, as usual, the lisa lab. Speaker: Cho KyungHyun Title: Neural Networks in Helsinki and Why I like RBMs Abstract: I will start the talk by briefly presenting research activities related to machine learning and neural networks going on in Helsinki, especially that of the department of information and computer science at the Aalto University. From there on I will explain my interest toward restricted Boltzmann machines and other unsupervised neural networks, which is based on the observation that neural networks may be viewed as an excellent machinery for approximating hierarchical probabilistic inference. In due course, I will introduce two novel approaches for training Boltzmann machines. At the end my current research direction focusing on recurrent neural networks and their potential application to machine translation will be discussed. Razvan

1 0

Tea-talk: Thursday, September 12 @ 1pm
by Razvan Pascanu 11 Sep '13

11 Sep '13

Please join us for tomorrow's tea-talk by Jason. The room will be, as usual, the lisa lab. Speaker: Jason Yosinski Title: Some Robots, Some Learning, and Some Ideas. Abstract: In this talk I'll give a brief overview of my past work in robotics and learning -- getting cars to drive themselves across deserts, letting robots teach themselves how to walk, and crowdsourcing design through human-in-the-loop objectives. I'll also discuss some recent thoughts I've had on deep learning and some research directions I think might prove fruitful. Hope to see many of you there, Razvan

1 0

Fwd: Rappel : Séminaire RALI-OLST du 26 juin 2013
by Yoshua Bengio 26 Jui '13

26 Jui '13

Hi all, For those who can follow a talk in French with slides in English, and with some interest in NLP, this should be an interesting talk: tomorrow at 11:30 at 3195. -- Yoshua Begin forwarded message: > From: Philippe Langlais <felipe(a)iro.umontreal.ca> > Subject: Fwd: Rappel : Séminaire RALI-OLST du 26 juin 2013 > Date: 25 juin 2013 21:58:25 HAE > To: Yoshua Bengio <bengioy(a)iro.UMontreal.CA> > > Yoshua, > > un séminaire qui pourrait intéresser certains membres de ta gang. > > Méthodes Statistiques en Traitement des Langues - Etat des lieux et perspectives. > François Yvon > Université Paris Sud et LIMSI/CNRS > Dans cet exposé, je vais présenter une rétrospective très générale de vingt années de travaux conduits en Traitement Automatique des Langue, en me focalisant exclusivement sur les méthodes fondées sur l'exploitation statistique de gros corpus annotés. En m'appuyant sur cette synthèse, je discuterai ensuite quelques perspectives pour le domaine du TAL, en discutant en particulier de ses interactions avec le domaine de la linguistique. > > (exposé en français, slides in English) > > f. > > > Début du message réexpédié : > >> De : Webmestre RALI <critter(a)iro.umontreal.ca> >> Date : 25 juin 2013 21:43:27 HAE >> À : rali-olst(a)iro.umontreal.ca >> Objet : Rappel : Séminaire RALI-OLST du 26 juin 2013 >> >> Annonce de séminaire RALI-OLST >> >> Méthodes Statistiques en Traitement des Langues - Etat des lieux et perspectives. >> François Yvon >> Université Paris Sud et LIMSI/CNRS >> >> Le 26 juin 2013 à 11h30 >> Salle 3195, Pavillon André-Aisenstadt >> >> ---- >> >> >> Consulter l'annonce du séminaire sur le Web : >> http://rali.iro.umontreal.ca/rali/?q=fr/node/1222/show/3567 > > RALI / DIRO phone: +1 (514) 343 61 11 ext: 47494 > Université de Montréal perso: +1 (514) 507 5314 > C.P. 6128 fax: +1 (514) 343 5834 > Succursale Centre-ville email: felipe(a)iro.umontreal.ca > Montréal (Québec), Canada, H3C 3J7 >

1 0

Fwd: Rappel : Séminaire RALI-OLST du 26 juin 2013
by Yoshua Bengio 26 Jui '13

26 Jui '13

Hi all, For those who can follow a talk in French with slides in English, and with some interest in NLP, this should be an interesting talk: tomorrow at 11:30 at 3195. -- Yoshua Begin forwarded message: > From: Philippe Langlais <felipe(a)iro.umontreal.ca> > Subject: Fwd: Rappel : Séminaire RALI-OLST du 26 juin 2013 > Date: 25 juin 2013 21:58:25 HAE > To: Yoshua Bengio <bengioy(a)iro.UMontreal.CA> > > Yoshua, > > un séminaire qui pourrait intéresser certains membres de ta gang. > > Méthodes Statistiques en Traitement des Langues - Etat des lieux et perspectives. > François Yvon > Université Paris Sud et LIMSI/CNRS > Dans cet exposé, je vais présenter une rétrospective très générale de vingt années de travaux conduits en Traitement Automatique des Langue, en me focalisant exclusivement sur les méthodes fondées sur l'exploitation statistique de gros corpus annotés. En m'appuyant sur cette synthèse, je discuterai ensuite quelques perspectives pour le domaine du TAL, en discutant en particulier de ses interactions avec le domaine de la linguistique. > > (exposé en français, slides in English) > > f. > > > Début du message réexpédié : > >> De : Webmestre RALI <critter(a)iro.umontreal.ca> >> Date : 25 juin 2013 21:43:27 HAE >> À : rali-olst(a)iro.umontreal.ca >> Objet : Rappel : Séminaire RALI-OLST du 26 juin 2013 >> >> Annonce de séminaire RALI-OLST >> >> Méthodes Statistiques en Traitement des Langues - Etat des lieux et perspectives. >> François Yvon >> Université Paris Sud et LIMSI/CNRS >> >> Le 26 juin 2013 à 11h30 >> Salle 3195, Pavillon André-Aisenstadt >> >> ---- >> >> >> Consulter l'annonce du séminaire sur le Web : >> http://rali.iro.umontreal.ca/rali/?q=fr/node/1222/show/3567 > > RALI / DIRO phone: +1 (514) 343 61 11 ext: 47494 > Université de Montréal perso: +1 (514) 507 5314 > C.P. 6128 fax: +1 (514) 343 5834 > Succursale Centre-ville email: felipe(a)iro.umontreal.ca > Montréal (Québec), Canada, H3C 3J7 >

1 0

ICML-practice talks: Thursday, June 13th, AA3195, @ 2:00PM
by Guillaume Desjardins 13 Jui '13

13 Jui '13

Please join us tomorrow afternoon (2:00-3:30PM) for a special round of ICML practice talks. Speakers will be Razvan Pascanu (remotely via Google Hangouts) and Yoshua Bengio (for both a workshop talk, and a 5 min ICML spotlight). Titles and abstract below. The talk will be held in AA3195 as usual. Speaker: Razvan Pascanu Title: On the difficulty of training Recurrent Neural Networks There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section. Speaker: Yoshua Bengio (workshop talk) Title: From Latent Anonymous Variables to Deep Stochastic Networks Graphical models with anonymous latent variables (such as Markov Random Fields and variants of Boltzmann machines) are very powerful models which have had a great impact in machine learning. We propose here to consider a type of models over the joint distribution of observed variables that shares many of the properties of these anonymous latent variable models, but without the need for potentially hurtful approximate inference or approximation of the partition function during training (or both, like in Deep Boltzmann Machines). The only approximation is function approximation. The proposed deep stochastic networks can provably estimate the underlying joint distribution of the data (or a conditional, if discriminant learning is preferred), in the sense that they are consistent estimators of this joint or conditional distribution. However, unlike the usual probabilistic models, they only represent the learned distribution implicitly, through the convergent distribution of a Markov chain. Like them, though, they can handle missing values and structured output. Unlike them, they can be trained by straightforward gradient descent in a deep (possibly recurrent) network in which noise is injected. Most of the impressive experimental progress in deep learning in recent years (in particular for speech and object recognition) has been with with deep supervised networks, sometimes with noise injected (dropout). Interestingly, the same kinds of architectures can be used to train the proposed unsupervised (or structured output) models. Speaker: Yoshua Bengio (ICML spotlight). Better Mixing via Deep Representations It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples.

3 2

Fwd: [Lisa_labo] John Langford presentation
by Yoshua Bengio 06 Jui '13

06 Jui '13

Hi, John Langford's presentation has been taped by Frederic Bastien and is ready for your enjoyment (the video and the slides). > https://www.youtube.com/user/nouiz -- Yoshua Begin forwarded message: > From: Frédéric Bastien <nouiz(a)nouiz.org> > Subject: [Lisa_labo] John Langford presentation > Date: 6 juin 2013 15:28:33 HAE > To: Lisa Labo <lisa_labo(a)iro.umontreal.ca> > Cc: Chris Pal <chris.j.pal(a)gmail.com>, John Langford <jl(a)hunch.net> > > Hi, > > There was problem on Mac with the previous video. Quicktime support just a subset of H264 codec. So I uploaded them on youtube. You can see them here (with links to the slides): > > https://www.youtube.com/user/nouiz > > The text on the projector isn't easy to read. It is like that in the original. You can check the slide to read them correctly. > > Frédéric > _______________________________________________ > Lisa_labo mailing list > Lisa_labo(a)iro.umontreal.ca > https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo

1 0

Talk by Anoop Korattikara: Thursday, June 6th at 2:00pm (AA3195)
by Guillaume Desjardins 06 Jui '13

06 Jui '13

Please join us this upcoming Thursday (June 6th) for the following talk given by Anoop Korattikara (student of Max Welling). The talk will be held at 2:00pm in AA3195 as usual. Title: Markov Chain Monte Carlo and the Bias-Variance Tradeoff Bayesian posterior sampling can be painfully slow on very large datasets, since traditional MCMC methods such as Hybrid Monte Carlo are designed to be asymptotically unbiased and require processing the entire dataset to generate each sample. Thus, given a small amount of sampling time, the variance of estimates computed using such methods could be prohibitive. We argue that lower risk estimates can often be obtained using ™approximate™ MCMC methods that mix very fast (and thus lower the variance quickly) at the expense of a small bias in the stationary distribution. I will first talk about two such biased algorithms: Stochastic Gradient Langevin Dynamics and its successor Stochastic Gradient Fisher Scoring, both of which use stochastic gradients estimated from mini-batches of data, allowing them to mix very fast. Then I will present our current work on a new (biased) MCMC algorithm that uses a sequential hypothesis test to approximate the Metropolis-Hastings test, allowing us to accept/reject samples with high confidence using only a fraction of the data required for the exact test.

2 1

Tea-talk: Thursday, April 11 @ 2pm (lisa lab)
by Guillaume Desjardins 10 Avr '13

10 Avr '13

Please join us for tomorrow's tea-talk, by our newest recruit Stephan, and Çağlar. The talk will be held from 2:00-3:00 pm in the lisa lab, as usual. The next two tea-talks slots (April 18th + 25th) will be reserved for ICLR rehearsal talks, so please ping me to schedule them if you haven't already. Speaker: Stephan Gouws Abstract/Bio: Stephan Gouws is a PhD research exchange student at UdeM from Stellenbosch University in South Africa. Besides learning French and hoping for "real" summer to finally roll around, he also works on applying neural language models to transfer-learning problems involving sequence-tagging models in natural language processing. An example might be to extract the names of companies, people and products from raw text drawn from two different domains, when labelled information is only available for one of the domains. Stephan's talk will serve as a quick introduction to who he is, where he's from, and a little more on what he is working on. Speaker: Çağlar Gülçehre Abstract: In our accepted ICLR 2013 paper(Knowledge Matters:Importance of prior information for optimization), we have shown that it is possible to solve really hard/abtract vision problems by incorporating hints at the intermediate level of representation and disentangling the factors of variations. In our followup research project, we aim to expand that approach to harder&possibly more abstract problems in both biologically and computationally more plausible way. We are still at the early stage of our project and the purpose of this talk is to get feedback and criticism on our research.

1 0

Tea-talk: Thursday, March 28th at 2:00pm, AA3195
by Guillaume Desjardins 28 Mar '13

28 Mar '13

We have another round of ICLR tea-talks tomorrow, at 2:00pm. Speakers and papers below. See you there ! I will be looking for 2 more volunteers for the following week, so please speak up if you haven't presented yet. Presenter: Li Yao "No More Pesky Learning Rates". Tom Schaul, Sixin Zhang, Yann LeCun "Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients<http://openreview.net/document/c14f2204-fd66-4d91-bed4-153523694041#c14f220…>". Tom Schaul, Yann LeCun Presenter: Mehdi Mirza Stochastic Pooling for Regularization of Deep Convolutional Neural Networks.<http://openreview.net/document/7230c92b-64e0-46f1-b6c8-d1c0018c5491#7230c92…>Matthew D. Zeiler, Rob Fergus -- Guillaume

1 1