- Lisa_teatalk - mailman.iro.umontreal.ca

Reminder: Tea Talk this morning!
by Aaron Courville 11 Jui '12

11 Jui '12

Just a reminder that we have a tea talk this morning at a special time: 11:00am. Details below Cheers, Aaron On Sat, Jun 9, 2012 at 9:21 AM, Aaron Courville <aaron.courville(a)gmail.com> wrote: > We will have Fred's postponed Tea Talk / Discussion on Monday (June > 11th) at 11:00am. This discussion relates to the questionnaire that > Fred sent out last week. Please be there. > > Time/Date: Monday (June 11th) at 11:00am > Location: LISA Lab. > > Title/Abstract: > > Title: Discussion of code development/lab admnistration/others > priority for the lab > > To help target our code development/administration/others, we will > discuss together the result of the questionary I emailed. > > > Cheers, > Aaron > > On Wed, Jun 6, 2012 at 11:12 AM, Guillaume Desjardins > <guillaume.desjardins(a)gmail.com> wrote: >> This tea-talk will unfortunately be post-poned, due to a conflict with >> the UdeM-McGill seminar. More details to follow. >> >> On Tue, Jun 5, 2012 at 10:49 PM, Aaron Courville >> <aaron.courville(a)gmail.com> wrote: >>> Hi, >>> >>> We are restarting our tea talk series after the NIPS break. This week >>> we have Frédéric leading a discussion of our code development >>> priorities in the lab. Hope to see you there. >>> >>> When: Thursday June 7th, 3:00pm >>> Where: LISA lab >>> >>> Title: Discussion of code development/lab admnistration/others >>> priority for the lab >>> >>> Point: To help target our code development/administration/others, we will >>> discuss together the result of the questionary I emailed. >>> >>> >>> Cheers, >>> Aaron >>> >>> -- >>> Aaron C. Courville >>> Département d’Informatique et >>> de recherche opérationnelle >>> Université de Montréal >>> email:Aaron.Courville@gmail.com >>> _______________________________________________ >>> Lisa_labo mailing list >>> Lisa_labo(a)iro.umontreal.ca >>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo > > > > -- > Aaron C. Courville > Département d’Informatique et > de recherche opérationnelle > Université de Montréal > email:Aaron.Courville@gmail.com -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Monday!
by Aaron Courville 09 Jui '12

09 Jui '12

We will have Fred's postponed Tea Talk / Discussion on Monday (June 11th) at 11:00am. This discussion relates to the questionnaire that Fred sent out last week. Please be there. Time/Date: Monday (June 11th) at 11:00am Location: LISA Lab. Title/Abstract: Title: Discussion of code development/lab admnistration/others priority for the lab To help target our code development/administration/others, we will discuss together the result of the questionary I emailed. Cheers, Aaron On Wed, Jun 6, 2012 at 11:12 AM, Guillaume Desjardins <guillaume.desjardins(a)gmail.com> wrote: > This tea-talk will unfortunately be post-poned, due to a conflict with > the UdeM-McGill seminar. More details to follow. > > On Tue, Jun 5, 2012 at 10:49 PM, Aaron Courville > <aaron.courville(a)gmail.com> wrote: >> Hi, >> >> We are restarting our tea talk series after the NIPS break. This week >> we have Frédéric leading a discussion of our code development >> priorities in the lab. Hope to see you there. >> >> When: Thursday June 7th, 3:00pm >> Where: LISA lab >> >> Title: Discussion of code development/lab admnistration/others >> priority for the lab >> >> Point: To help target our code development/administration/others, we will >> discuss together the result of the questionary I emailed. >> >> >> Cheers, >> Aaron >> >> -- >> Aaron C. Courville >> Département d’Informatique et >> de recherche opérationnelle >> Université de Montréal >> email:Aaron.Courville@gmail.com >> _______________________________________________ >> Lisa_labo mailing list >> Lisa_labo(a)iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Thursday
by Aaron Courville 06 Jui '12

06 Jui '12

Hi, We are restarting our tea talk series after the NIPS break. This week we have Frédéric leading a discussion of our code development priorities in the lab. Hope to see you there. When: Thursday June 7th, 3:00pm Where: LISA lab Title: Discussion of code development/lab admnistration/others priority for the lab Point: To help target our code development/administration/others, we will discuss together the result of the questionary I emailed. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

2 1

Tea Talk this week!
by Aaron Courville 07 Mai '12

07 Mai '12

This week we will have a tea talk by Pascal Vincent. Hope to see you all there. When: Thursday May 10th, 15h00 Where: LISA Lab. Title: Biasing estimation towards Manifold prior hypothesis Abstract: I will present the fruits of my reflections on the manifold hypothesis, starting form the mere geometric intuition, and showing how it may be translated into specific regularization schemes when fitting energy based models or auto-encoders. I will contrast the resulting regularized parameter estimation schemes with what is implied by maximum likelihood, score matching, denoising auto-encoder and contractive auto-encoder criteria. The alternative regularization terms I propose should aim at modeling actual low dimensional manifolds, irrespective of model parametrization and contrary to our current approaches. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

No Tea Talk Today. (EOM)
by Aaron Courville 03 Mai '12

03 Mai '12

-- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Thursday!
by Aaron Courville 24 Avr '12

24 Avr '12

Hey, The Tea Talk this week will be given by Yoshua Bengio, discussing his latest ideas regarding density estimation with CAE-type models. When: 15h00, Thursday, April 26th. Where: LISA Lab. Title/Abstract: Density Function Induction by Estimating Local Density Structure I am exploring a novel learning framework for unsupervised learning which should be viewed as an alternative to maximum likelihood training, score matching, pseudo-likelihood, etc. It may help to justify algorithms such as the Contracting Auto-Encoder and clarify the corresponding density model learned. The basic information learned by a CAE and a DAE are (1) where to move in input space so as to increase density (i.e. towards the reconstruction), and (2) if near a manifold (i.e. a mode of dimension > 0) in what directions (and how much) to move in order to remain in a high-density region. We have shown that one can use geometric intuition to define a sampling algorithm in the form of an MCMC using this information, and that it mixes rather well and produces visually and numerically good samples. I first show that this implicitly defines a density learned by the model, which is the asymptotic distribution of that MCMC. I then discuss consistency, i.e., what it is that (1) and (2) above should be such that this learned implicit density function corresponds (at least asymptotically) to the unknown generating process from which the training data were sampled. This is work in progress with yet many mysteries and open questions. In addition to providing a more solid probabilistic footing for CAE-like or DAE-like algorithms this work could potentially yield better training algorithms for this type of learner as well as better sampling algorithms for them. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Tomorrow!
by Aaron Courville 19 Avr '12

19 Avr '12

Welcome the return of the tea talk series (well for now anyway)! This week we have Guillaume Desjardins letting us about some of his resent insights into subspace auto-encoders. Hope to see you there. When: Thursday, April 19th (tomorrow): 15h00 Where: LISA Lab (AA3256) Title: Subspace Auto-Encoders In an ssRBM, each hidden unit h_i is invariant to the subspace spanned by the filters being pooled over. This appears to be a more powerful (and general ?) form of invariance than traditional sum, max or mean pooling. Additionally, it has the benefit that while h_i gains a degree of invariance, knowledge about the particular configuration of the input remains accessible through the slab variables, which can be interpreted as coordinates within this subspace. This view motivates a novel approach to pooling in the context of auto-encoders. I will thus present the subspace auto-encoder framework, which replaces slab variables by linear projections onto filter sets and where hidden units are made sensitive to the norm of these projections. I will discuss its relationship to existing methods in the literature, as well as issues encountered so far. Time allowing, I will also discuss various alternative formulations which have been discussed with Pascal and Aaron. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Tomorrow!
by Aaron Courville 14 Mar '12

14 Mar '12

Hi All, Our regularly scheduled tea talk this week with be given by Nicolas Boulanger-Lewandowski. Nicolas will tell about some of his recent work. Hope to see you all there. When: 15h00, Thursday March 15th, 2012 Where: LISA lab (AA3256) Title: Discriminative non-negative matrix factorization for polyphonic music transcription Non-negative matrix factorization (NMF) is an unsupervised technique to decompose data into additive-only components. It is useful to infer interpretable recurring note events in a power spectrogram, or facial features in emotion recognition data. After reviewing recently developed models for polyphonic music generation, I will use NMF to transcribe audio excerpts into musical notation. I will introduce two discriminative criteria, both in the latent variable and autoencoder perspectives, to influence the NMF decomposition and improve the resulting transcription. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Tomorrow!
by Aaron Courville 11 Mar '12

11 Mar '12

Hi, Tomorrow we have a special Tea Talk at a special Time. A visitor, Pardis Noorzad, will present some of his work with Prof. Bob L. Sturm. Hope to see you there. When: Tomorrow, Monday March 12th, 2012 Where: TBA "SPARROW: Sparse approximation weighted regression" Abstract: We propose sparse approximation weighted regression (SPARROW), which is a nonparametric method of regression that takes advantage of the sparse linear approximation of a query point. SPARROW employs weights based on sparse approximation in the context of locally constant, locally linear, and locally quadratic regression to generate better estimates than for e.g., k-nearest neighbor regression and more generally, kernel-weighted local polynomial regression. Our experimental results show that SPARROW performs competitively. This is joint work with Prof. Bob L. Sturm. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 1

Tea Talk Tomorrow!
by Aaron Courville 07 Mar '12

07 Mar '12

Greetings, This week Razvan Pascanu will tell us about some of his recent work on optimization methods. Hope to see you there. When: 15h00 Tomorrow (Thursday March 8th 2012) Where: LIDA Lab, (AA3256) Title : Yet Another Optimization Technique for Machine Learning ? Abstract: I've been trying to understand what Hessian Free or related methods are doing, how and why they can claim to be getting better generalization error. Why they work for RNNs better then SGD in certain cases? I don't have all the answers, but I found a set of misnomers IMHO, and interesting connections between natural gradient and recently proposed methods. I want to provide a different interpretation of how fancier optimization should be done for complex non-linear models, and I want in the end to outline how one would go about improving proposed algorithms. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Tomorrow (NOTE: new location)
by Aaron Courville 01 Mar '12

01 Mar '12

After the ICML rush, we are restarting our tea talk series with some deep thoughts from Yoshua. See you there. When: 15h00 Thursday, March 1st. 2012 Where: AA1355 (NEW LOCATION) Title/Abstract: Evolving Culture vs Local Minima Yoshua Bengio We propose a theory that relates difficulty of learning in deep architectures to culture and language. It is articulated around the following hypotheses: (1) learning in an individual human brain is hampered by the presence of effective local minima; (2) this optimization difficulty is particularly important when it comes to learning higher-level abstractions, i.e., concept that cover a vast and highly-nonlinear span of sensory configurations; (3) such high-level abstractions are best represented in brains by the composition of many levels of representation, i.e., by deep architectures; (4) a human brain can learn such high-level abstractions if guided by the signals produced by other humans, which act as hints or indirect supervision for these high-level abstractions; and (5), language and the combination of old ideas into new ideas provide an efficient evolutionary recombination operator, and this allows rapid search in the space of communicable ideas that help humans build up better high-level internal representations of their world. These hypotheses put together imply that human culture and the evolution of ideas have been crucial to counter an optimization difficulty: this optimization difficulty would otherwise make it very difficult for human brains to capture high-level knowledge of the world. The theory is grounded in experimental observations of the difficulties of training deep artificial neural networks. Plausible consequences of this theory for the efficiency of cultural evolutions are sketched. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

3 2

Re: [Lisa_teatalk] [Lisa_labo] Tea Talk Tomorrow (NOTE: new location)
by Yoshua Bengio 01 Mar '12

01 Mar '12

Note the unusual location! If you participate in the pylearn2 code sprint, please take a refreshing break at 3pm... Also, for those interested, I am writing a paper on this subject. The draft is in articles/2012/culture_vs_localmin/ (and I attach a pdf of the latest version for your convenience) and I am supposed to submit it this week to as a book chapter, so any comment is welcome. Note also that this is meant to be the starting point for Caglar's thesis (who joined us this term for a PhD). -- Yoshua On 2012-02-29, at 15:11, Aaron Courville wrote: > After the ICML rush, we are restarting our tea talk series with some > deep thoughts from Yoshua. See you there. > > When: 15h00 Thursday, March 1st. 2012 > Where: AA1355 (NEW LOCATION) > > Title/Abstract: > > Evolving Culture vs Local Minima > Yoshua Bengio > > We propose a theory that relates difficulty of learning in deep > architectures to culture and language. It is articulated around the > following hypotheses: > > (1) learning in an individual human brain is hampered by the presence > of effective local minima; > (2) this optimization difficulty is particularly important when it > comes to learning higher-level abstractions, i.e., concept that cover > a vast and highly-nonlinear span of sensory configurations; > (3) such high-level abstractions are best represented in brains by the > composition of many levels of representation, i.e., by deep > architectures; > (4) a human brain can learn such high-level abstractions if guided by > the signals produced by other humans, which act as hints or indirect > supervision for these high-level abstractions; and > (5), language and the combination of old ideas into new ideas provide > an efficient evolutionary recombination operator, and this allows > rapid search in the space of communicable ideas that help humans build > up better high-level internal representations of their world. > > These hypotheses put together imply that human culture and the > evolution of ideas have been crucial to counter an optimization > difficulty: this optimization difficulty would otherwise make it very > difficult for human brains to capture high-level knowledge of the > world. The theory is grounded in experimental observations of the > difficulties of training deep artificial neural networks. Plausible > consequences of this theory for the efficiency of cultural evolutions > are sketched. > > > Cheers, > Aaron > > -- > Aaron C. Courville > Département d’Informatique et > de recherche opérationnelle > Université de Montréal > email:Aaron.Courville@gmail.com > _______________________________________________ > Lisa_labo mailing list > Lisa_labo(a)iro.umontreal.ca > https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo

1 0

Tea Talk this Week!
by Aaron Courville 28 Fév '12

28 Fév '12

After the ICML rush, we are restarting our tea talk series with some deep thoughts from Yoshua. See you there. When: 15h00 Thursday, March 1st. 2012 Where: TBA Title/Abstract: Evolving Culture vs Local Minima Yoshua Bengio We propose a theory that relates difficulty of learning in deep architectures to culture and language. It is articulated around the following hypotheses: (1) learning in an individual human brain is hampered by the presence of effective local minima; (2) this optimization difficulty is particularly important when it comes to learning higher-level abstractions, i.e., concept that cover a vast and highly-nonlinear span of sensory configurations; (3) such high-level abstractions are best represented in brains by the composition of many levels of representation, i.e., by deep architectures; (4) a human brain can learn such high-level abstractions if guided by the signals produced by other humans, which act as hints or indirect supervision for these high-level abstractions; and (5), language and the combination of old ideas into new ideas provide an efficient evolutionary recombination operator, and this allows rapid search in the space of communicable ideas that help humans build up better high-level internal representations of their world. These hypotheses put together imply that human culture and the evolution of ideas have been crucial to counter an optimization difficulty: this optimization difficulty would otherwise make it very difficult for human brains to capture high-level knowledge of the world. The theory is grounded in experimental observations of the difficulties of training deep artificial neural networks. Plausible consequences of this theory for the efficiency of cultural evolutions are sketched. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

No tea talk this week.
by Aaron Courville 21 Fév '12

21 Fév '12

Due to the ICML rush, there will be no tea talk this week. Tea talks will start up again next week (March 1st) with a talk by Yoshua. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Today!
by Aaron Courville 16 Fév '12

16 Fév '12

Hi Gang, Today we will have a our new post-doc, Heng Luo, tell us about his previous work. Looking forward to seeing you all there. When: Today, 15h00 Where: LISA Lab (AA3256) Title: "Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM". Abstract: Restricted Boltzmann Machines are commonly used in unsupervised learning to extract features from training data. Since these features are learned for regenerating training data a classifier based on them has to be trained. If only a few of the learned features are discriminative other non-discriminative features will distract the classifier during the training process and thus waste computing resources for testing. In this paper, we present a hybrid third-order Restricted Boltzmann Machine in which class-relevant features (for recognizing) and class-irrelevant features (for generating only) are learned simultaneously. As the classification task uses only the class-relevant features, the test itself becomes very fast. We show that class-irrelevant features help class-relevant features to focus on the recognition task and introduce useful regularization effects to reduce the norms of class-relevant features. Thus there is no need to use weight-decay for the parameters of this model. Experiments on the MNIST, NORB and Caltech101 Silhouettes data sets show very promising results. If time permits, I will also briefly discuss my paper, "Sparse Group Restricted Boltzmann Machines". Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk today
by Aaron Courville 09 Fév '12

09 Fév '12

Hi, Today we have our last tea talk in the best-of-NIPS series. See you there Where: LISA Lab (AA3256) When: 15h00 Who's talking: 1) Yann Dauphin 2) Xavier Glorot 3) Salah Rifai Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk today!
by Aaron Courville 02 Fév '12

02 Fév '12

Greetings, Today we have the next installment of the best-of-NIPS tea talk series. Looks like a great line-up. See you there. When: Today, 15h00 Where: LISA Lab. Speakers / Papers: 1) Nicolas Boulanger-Lewandowski "How Do Humans Teach: On Curriculum Learning and Teaching Dimension" 2) Grégoire "Group Invariant Scattering" only Mallat can do it. 3) Olivier Delalleau "Collaborative Learning of Preferences for Recommending Games and Media" by Thore Graepel Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Today!
by Aaron Courville 26 Jan '12

26 Jan '12

Today we continue our best-of-NIPS tea talk series. See you there. When: 15h00 today! Where: AA3256 (LISA LAB) On tap: 1) Ian Goodfellow Beyond spatial pyramids 2) David Warde-Farley TBD 3) Arnaud Bergeron Distributed Delayed Stochastic Optimization -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Today!
by Aaron Courville 19 Jan '12

19 Jan '12

All, Today, we have our first tea talk of 2012. It's the first of our "Best-of-NIPS" series. For those of you who haven't signed up yet (you know who you are) please do so on the Google doc: https://docs.google.com/spreadsheet/ccc?key=0As6zk-mwfSMQdDdlNHdtZExxNjBYaX… IF YOU WENT TO NIPS, YOU SHOULD BE ON THIS LIST. Today's talk will be at 15h00 in the LISA lab. Speakers and papers: 1) Aaron Courville "Selecting Receptive Fields in Deep Networks" by Coates and Ng 2) Razvan Pascanu "Krylov Subspace Descent for Deep Learning" by Oriol Vinyals and Daniel Povey 3) Guillaume Desjardins unsure, but possibly "Efficient coding of natural images with a population of noisy Linear-Nonlinear neurons" Hope to see you there. Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk 2012 Kickoff
by Aaron Courville 13 Jan '12

13 Jan '12

Hi Gang, Following tradition, we will begin our Tea Talks of 2012 with a best-of-NIPS series. If you went to NIPS2011 you are *strongly encouraged* to present a paper. Talks should be about 20 minutes long (3 per Tea Talk). Please try to pick a paper that you feel will be relevant and interesting to our group. If you are unsure about the suitability of a paper, you are encouraged to talk to Yoshua, Pascal or me (Aaron) for advice. Sign up on the Google doc: https://docs.google.com/spreadsheet/ccc?key=0As6zk-mwfSMQdDdlNHdtZExxNjBYaX… Also, those who sign up for next week will have my sincere gratitude. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 1

Tea Talk schedule for this term
by Aaron Courville 09 Jan '12

09 Jan '12

I'm trying to confirm a time for the tea talks this term. The proposal is for them to take place Thursdays at 3pm. Please let me know if you have a significant conflict with this time slot (e.g. the time conflicts with a course you plan to attend). Also, I'm looking for speakers for the new term. So if you'd like to give a talk in the next few weeks please let me know. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk Tomorrow!
by Aaron Courville 07 Déc '11

07 Déc '11

Hi gang, This week we have two more practice talks for the upcoming NIPS conference. Ian and Guillaume (take 2) will be presenting their recent work. Come with questions, comments and critiques. When: Wednesday Dec. 7nd, 14h00 Where: AA3195 Speaker: Ian Goodfellow Title: Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery We introduce spike-and-slab sparse coding (S3C), an unsupervised feature discovery algorithm. S3C is based on a generative model that resembles both the spike-and-slab RBM and sparse coding. Since exact inference in this model is intractable, we derive a structured variational inference procedure and employ a variational EM training algorithm. We demonstrate that this approach improves upon the supervised learning capabilities of both sparse coding and the ssRBM on the CIFAR-10 dataset. We evaluate our approach’s potential for semi-supervised learning on subsets of CIFAR-10. We demonstrate state-of-the art self-taught learning performance on the STL-10 dataset and use our method to win the NIPS 2011 Workshop on Challenges In Learning Hierarchical Models’ Transfer Learning Challenge. +++++++++++++++++++++++++++++++++++++++++++++++++++++ Speaker: Guillaume Desjardins Title: On Tracking The Partition Function Markov Random Fields (MRFs) have proven very powerful both as density estimators and feature extractors for classification. However, their use is often limited by an inability to estimate the partition function $Z$. In this paper, we exploit the gradient descent training procedure of restricted Boltzmann machines (a type of MRF) to {\bf track} the log partition function during learning. Our method relies on two distinct sources of information: (1) estimating the change $\Delta Z$ incurred by each gradient update, (2) estimating the difference in $Z$ over a small set of tempered distributions using bridge sampling. The two sources of information are then combined using an inference procedure similar to Kalman filtering. Learning MRFs through Tempered Stochastic Maximum Likelihood, we can estimate $Z$ using no more temperatures than are required for learning. Comparing to both exact values and estimates using annealed importance sampling (AIS), we show on several datasets that our method is able to accurately track the log partition function. In contrast to AIS, our method provides this estimate at each time-step, at a computational cost similar to that required for training alone. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Reminder: Tea Talk Today!
by Aaron Courville 02 Déc '11

02 Déc '11

hi gang, This week we have a practice talk for the upcoming NIPS conference. Arnaud will give us his best pitch. Come with questions, comments and critiques. When: Firday Dec. 2nd, 14h00 Where: AA1360 Speaker: Arnaud Bergeron A Common GPU n-Dimensional Array for Python and C Currently there are multiple incompatible array/matrix/n-dimensional base object implementations for GPUs. This hinders the sharing of GPU code and causes duplicate development work. This paper proposes and presents a first version of a common GPU n-dimensional array (tensor) named GpuNdArray that works with both CUDA and OpenCL. It will be usable from Python, C, and possibly other programming languages. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Tea Talk this FRIDAY!
by Aaron Courville 30 Nov '11

30 Nov '11

hi gang, This week we have two practice talks (well one practice talk and one practice poster :-)) for the upcoming NIPS conference. Guillaume and Arnaud will each have about 30 minutes to give us there best pitch. Come with questions, comments and critiques. When: Firday Dec. 2nd, 14h00 Where: TBA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Speaker: Guillaume Desjardins Title: On Tracking The Partition Function Markov Random Fields (MRFs) have proven very powerful both as density estimators and feature extractors for classification. However, their use is often limited by an inability to estimate the partition function $Z$. In this paper, we exploit the gradient descent training procedure of restricted Boltzmann machines (a type of MRF) to {\bf track} the log partition function during learning. Our method relies on two distinct sources of information: (1) estimating the change $\Delta Z$ incurred by each gradient update, (2) estimating the difference in $Z$ over a small set of tempered distributions using bridge sampling. The two sources of information are then combined using an inference procedure similar to Kalman filtering. Learning MRFs through Tempered Stochastic Maximum Likelihood, we can estimate $Z$ using no more temperatures than are required for learning. Comparing to both exact values and estimates using annealed importance sampling (AIS), we show on several datasets that our method is able to accurately track the log partition function. In contrast to AIS, our method provides this estimate at each time-step, at a computational cost similar to that required for training alone. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Speaker: Arnaud Bergeron A Common GPU n-Dimensional Array for Python and C Currently there are multiple incompatible array/matrix/n-dimensional base object implementations for GPUs. This hinders the sharing of GPU code and causes duplicate development work. This paper proposes and presents a first version of a common GPU n-dimensional array (tensor) named GpuNdArray that works with both CUDA and OpenCL. It will be usable from Python, C, and possibly other programming languages. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0

Reminder TEA TALK TODAY, 13h30 @ AA1207
by Aaron Courville 23 Nov '11

23 Nov '11

Greeting people of LISA, Tomorrow we have Yann Dauphin practising his NIPS oral. Be there. NOTE: new time and new location. When: 13h30 Wednesday Nov. 23. Where: AA1207. The title is "The Manifold Tangent Classifier". Maybe you could also mention it's a practice talk for NIPS. Abstract: We combine three important ideas present in previous work for building classifiers: the semi-supervised hypothesis (the input distribution contains information about the classifier), the unsupervised manifold hypothesis (data density concentrates near low-dimensional manifolds), and the manifold hypothesis for classification (different classes correspond to disjoint manifolds separated by low density). We exploit a new algorithm for capturing manifold structure (high-order contractive autoencoders) and we show how it builds a topological atlas of charts, each chart being characterized by the principal singular vectors of the Jacobian of a representation mapping. This representation learning algorithm can be stacked to yield a deep architecture, and we combine it with a domain knowledge-free version of the TangentProp algorithm to encourage the classifier to be insensitive to local directions changes along the manifold. Record-breaking results are obtained and we find that the learned tangent directions are very meaningful. Cheers, Aaron -- Aaron C. Courville Département d’Informatique et de recherche opérationnelle Université de Montréal email:Aaron.Courville@gmail.com

1 0