- Lisa_teatalk - mailman.iro.umontreal.ca

Fri 15th: ICLR practice talks (tomorrow!)
by Jörg Bornschein 14 Avr '16

14 Avr '16

Hi, this is a last minute announcement for our tea talk tomorrow: We will have David Krueger and Zhouhan Lin present their ICLR papers. These are practice talks for their presentations -- so it would be great to have many of you there and to get a lot of constructive feedback. When: Fri 15th, 14:30 Where: AA3195 Regularizing RNNs by Stabilizing Activations David Krueger, Roland Memisevic http://arxiv.org/abs/1511.08400 We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states' norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modelling and phoneme recognition, and outperforming weight noise and dropout. We set state of the art (17.5% PER) for an RNN on the TIMIT phoneme recognition task, without using beam-search. With this penalty term, IRNN can achieve similar performance to LSTM on language modelling, although adding the penalty term to the LSTM results in superior performance. Our penalty term also prevents the exponential growth of IRNN's activations outside of their training horizon, allowing them to generalize to much longer sequences. Neural Networks with Few Multiplications Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, Yoshua Bengio http://arxiv.org/abs/1510.03009 For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks. Hope to see you tomorrow, Jorg

2 1

Tea Talk: Mariana -- The Cutest Deep Learning Framework
by Jörg Bornschein 12 Avr '16

12 Avr '16

Hi, this Friday we will have Tariq Daouda present the deep learning framework "Mariana": --- Title: Mariana -- The Cutest Deep Learning Framework Who: Tariq Daouda When: Fri April 8th, 14:30 Where: AA3195 Abstract: As neural nets increase in complexity they also become harder to write and harder to teach. Our hypothesis is that these difficulties stem from the absence of a language that elegantly describe neural networks. Mariana (named after the deepest place on earth, the Mariana trench) is an attempt to create such a language within python. That being said, you can also call it an Extendable Python Machine Learning Framework build on top of Theano that focuses on ease of use. best, j

2 2

TeaTalk by Çağlar Gülçehre: Noisy Activation Functions
by Jörg Bornschein 30 Mar '16

30 Mar '16

Hi, this Friday we'll have a talk by Çağlar Gülçehre about Noisy Activation Functiuons: Who: Çağlar Gülçehre When: Fri. April 1st, 14:30 Where: AA 3195 Common non-linear activation functions used in neural networks can cause training difficulties due to the saturation behaviour of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent to explore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results. best and looking forward to see you there, Jorg Bornschein

2 1

Talk Fri. 19th: Towards bridging the gap between deep learning and biology
by Jörg Bornschein 19 Fév '16

19 Fév '16

Hi everyone, this weeks tea talk will be a bit special -- Yoshua Bengio will talk about bridging the gap between deep learning and biology; and we will have a Dutch TV crew around to film how "science is done". I expect it will be very crowded -- so maybe come a few minutes earlier. -- Title: Towards bridging the gap between deep learning and biology Who: Yoshua Bengio When: 14:30 Where: AA 3195 We explore the following crucial question: how could brains potentially perform the kind of powerful credit assignment that allows hidden layers of a very deep network to be trained and that has been so successful with backprop in deep nets recently? Global reinforcement learning signals have too much variance (scaling with the number of neurons or synapses) to be credible from a machine learning point of view. Concerns have been raised about how something like back-propagation could be implemented in brains. We present several intriguing results all aimed at answering this question and possibly providing pieces of this puzzle. We start with an update rule that yields updates similar to STDP but that is anchored in quantities such as pre-synaptic and post-synaptic firing rates and temporal rates of change. We then show that if neurons are connected symmetrically (with feedback connections) and define an energy function, (a) their behaviour corresponds both to inference, i.e., going down the energy, and propagating error gradients, (b) after a prediction is made on a sensor and an actual value is observed, the early phases of inference in this network actually propagate prediction error gradients, and (c) using the above STDP-inspired rule yields a gradient descent step on prediction error for the fixed point of the recurrent network (d) contrary to previously believed for such fixed-point networks, it is not necessary to do a full relaxation in the positive phase (perturbation propagation does the backprop job). Finally, we discuss some of the open problems we are facing to move forward, such as avoiding the negative phase fixed point relaxation (just like we got rid of the positive phase one), avoiding the forced symmetry of synaptic weights, the question of learning the full joint distribution and not just a point prediction, doing unsupervised learning, and handling time. -- Best and looking forward to see you there j

2 3

Tea Talks 12th Feb
by Jörg Bornschein 08 Fév '16

08 Fév '16

Hi everyone, besides of David Duvenauds talk on Thursday, we will have two tea talks this Friday: First Sixin Zhang will talk about learning with elastic averaging SGD and afterwards James Henderson will talk about Spike Event Based Learning in Neural Networks Here are the details: Title: Deep learning with Elastic Averaging SGD Who: Sixin Zhang When: Fri 12th Feb, 14:30 Where: AA 3195 In this talk, I'll introduce Elastic Averaging SGD (EASGD). It is a stochastic optimization method developed to train large deep learning models in the parallel computing environment under communication constraints. We start from the basic ideas of SGD and Average SGD to motivate EASGD. The basic stability and averaging properties of synchronous EASGD will be discussed. Then we accelerate EASGD with Nesterov's optimal method, giving EAMSGD. The robustness of EASGD and EAMSGD to the communication delay allows us to train CIFAR-10 and ImageNet-1k image classification tasks with relatively large parameter communication period. We will try to understand why the Downpour method does not work so well, and see how to fix that. If time permits, I'll discuss the tree extension of EASGD, and why it's a good or maybe a bad idea. This is joint-work with Anna Choromanska and Yann LeCun. Title: Spike Event Based Learning in Neural Networks Who: James Henderson When: Fri 12th, 15:30 Where: AA 3195 A scheme is derived for learning connectivity in spiking neural networks. The scheme learns instantaneous firing rates that are conditional on the activity in other parts of the network. The scheme is independent of the choice of neuron dynamics or activation function, and network architecture. It involves two simple, online, local learning rules that are applied only in response to occurrences of spike events. This scheme provides a direct method for transferring ideas between the fields of deep learning and computational neuroscience. This learning scheme is demonstrated using a layered feedforward spiking neural network trained self-supervised on a prediction and classification task for moving MNIST images collected using a Dynamic Vision Sensor.

1 0

Talk: David Duvenaud on Gradient-based hyperparameter optimization through reversible learning
by Jörg Bornschein 04 Fév '16

04 Fév '16

Hi, next *Thursday*, Feb. 11th at 3:30 pm we will have David Duvenaud give a talk at the department colloquium. Title: Gradient-based hyperparameter optimization through reversible learning Who: David Duvenaud When: Thursday, Feb. 11th; 3:30 pm Where: AA 3195 Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. This lets us optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural net architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum. We'll also discuss related applications to nonlinear filtering in model-based reinforcement learning. Looking forward to see you there, j

1 0

TeaTalk: Discriminative Regularizers and Metrics for Generative Models
by Jörg Bornschein 19 Jan '16

19 Jan '16

Hi, This Friday Alex Lamb will talk about ongoing work and cool new results related to generative modeling: When: Friday, 22th Jan, 14:30-15:30 Where: Z-209 Pavilion Mc Nicoll Who: Alex Lamb Title: Discriminative Regularizers and Metrics for Generative Models DeepDream and DeepStyle have shown that the representations learned by discriminatively trained classifiers can be used to produce photorealistic transformations of images. In this talk, we explore the question of whether these discriminatively learned representations can be used to enhance the quality of generative models. Our conjecture is that labels correspond to characteristics of natural data which are the most salient to humans: identity in faces, objects in images, and utterances in speech. We explore the use of this idea as both a regularizer and an evaluation metric for generative models and present preliminary experimental results on SVHN and CIFAR. See you there, j

1 0

Talk: Spike Event Based Learning in Neural Networks
by Jörg Bornschein 18 Jan '16

18 Jan '16

Hi, we are trying to find a time slot for James Handerson to give a talk about Spike Event Based Learning in Neural Networks from a ML perspective ( http://arxiv.org/abs/1502.05777) The usual tea-talk slot is already taken, so please participate in the doodle http://doodle.com/poll/2myxur4wv3us6q3q Merci, j

1 0

Talk on Friday: Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation
by Jörg Bornschein 15 Jan '16

15 Jan '16

Hi everyone, I'd like to announce the first Tea Talk in 2016: (BTW, Happy New Year, to those I didn't talk to yet!) Sina Honari will talk about "Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation" -- When: Fri., Jan. 15th 2016, 14:30 Who: Sina Honari Where: AA3195 (to be confirmed) Link: http://arxiv.org/abs/1511.07356 Abstract: Deep neural networks with alternating convolutional, max-pooling and decimation layers are widely used in state of the art architectures for computer vision. Max-pooling purposefully discards precise spatial information in order to create features that are more robust, and typically organized as lower resolution spatial feature maps. On some tasks, such as whole-image classification, max-pooling derived features are well suited; however, for tasks requiring precise localization, such as pixel level prediction and segmentation, max-pooling destroys exactly the information required to perform well. Precise localization may be preserved by shallow convnets without pooling but at the expense of robustness. Can we have our max-pooled multi-layered cake and eat it too? Several papers have proposed summation and concatenation based methods for combining upsampled coarse, abstract features with finer features to produce robust pixel level predictions. Here we introduce another model --- dubbed Recombinator Networks --- where coarse features inform finer features early in their formation such that finer features can make use of several layers of computation in deciding how to use coarse features. The model is trained once, end-to-end and performs better than summation-based architectures, reducing the error from the previous state of the art on two facial keypoint datasets, AFW and AFLW, by 30% and beating the current state-of-the-art on 300W without using extra data. We improve performance even further by adding a denoising prediction model based on a novel convnet formulation. -- Looking forward to see you there! j

2 3

Reminder: Talk at 2:30pm in room AA1207
by Jörg Bornschein 18 Déc '15

18 Déc '15

Just a friendly reminder: This Friday we will have Simon Lacoste-Julien, who is a researcher at INRIA in the SIERRA project team which is part of the Computer Science Department of École Normale Supérieure in Paris give a talk about Frank-Wolfe Optimization for Structured Machine Learning. Looking forward to see many of you there, j -- Title: Frank-Wolfe Optimization for Structured Machine Learning Who: Simon Lacoste-Julien Where: AA1207 When: Friday, 18th December, 2:30pm Abstract: The Frank-Wolfe (FW) optimization algorithm has lately re-gained popularity thanks in particular to its ability to nicely handle the structured constraints appearing in machine learning applications. However, its convergence rate is known to be slow (sublinear) when the solution lies at the boundary. In the first part of the talk, I will present some less well-known variants of the FW algorithm for which we proved their global linear convergence rate recently for the first time, highlighting at the same time an interesting geometric notion of "condition number" for the constraint set appearing in the constant. In the second part of the talk, I will present an application of these variants for approximate marginal inference in a Markov random field, by optimizing the TRW variational objective over the marginal polytope. The proposed algorithm, called "barrier FW" due to its similarities with barrier methods in optimization, is the first provably convergent algorithm of the TRW objective over the marginal polytope, and gives more accurate marginals than previous methods in our experiments. If time permits, I will also present how FW can be used to obtain adaptive quadrature rules and be used in particular in a particle filter to obtain better accuracy than the usual random sampling. This is joint work with David Sontag (NYU), Rahul Krishnan (NYU), Martin Jaggi (ETH), Fredrik Lindsten (U of Cambridge) and Francis Bach (INRIA).

1 0

Talk on Friday 18th, 2:30pm: Frank-Wolfe Optimization for Structured Machine Learning
by Jörg Bornschein 18 Déc '15

18 Déc '15

Hi everybody, this Friday we will have Simon Lacoste-Julien, who is a researcher at INRIA in the SIERRA project team which is part of the Computer Science Department of École Normale Supérieure in Paris give a talk about Frank-Wolfe Optimization for Structured Machine Learning. Looking forward to see many of you there, j -- Title: Frank-Wolfe Optimization for Structured Machine Learning Who: Simon Lacoste-Julien Where: AA3195 When: Friday, 18th December, 2:30pm Abstract: The Frank-Wolfe (FW) optimization algorithm has lately re-gained popularity thanks in particular to its ability to nicely handle the structured constraints appearing in machine learning applications. However, its convergence rate is known to be slow (sublinear) when the solution lies at the boundary. In the first part of the talk, I will present some less well-known variants of the FW algorithm for which we proved their global linear convergence rate recently for the first time, highlighting at the same time an interesting geometric notion of "condition number" for the constraint set appearing in the constant. In the second part of the talk, I will present an application of these variants for approximate marginal inference in a Markov random field, by optimizing the TRW variational objective over the marginal polytope. The proposed algorithm, called "barrier FW" due to its similarities with barrier methods in optimization, is the first provably convergent algorithm of the TRW objective over the marginal polytope, and gives more accurate marginals than previous methods in our experiments. If time permits, I will also present how FW can be used to obtain adaptive quadrature rules and be used in particular in a particle filter to obtain better accuracy than the usual random sampling. This is joint work with David Sontag (NYU), Rahul Krishnan (NYU), Martin Jaggi (ETH), Fredrik Lindsten (U of Cambridge) and Francis Bach (INRIA).

1 1

Talk by Yves Grandvalet on Tuesday, 15th December, 11am
by Jörg Bornschein 14 Déc '15

14 Déc '15

Hi, we are going to have Yves Grandvalet visiting on Monday 14th and Tuesday 15th, i.e. just after NIPS. Yves is the head of the Heudiasyc lab (where Antoine Bordes and Nicolas Usunier worked before they joined Facebook), and a past collaborator of LISA (he spent a few months at LISA a couple of years ago). He's would also like to discuss possible collaboration avenues between our labs (maybe just after his talk). ------------------------------------------------------------------------------------ Title : Beyond Support in Two-Stage Variable Selection Who: Yves Grandvalet When: Tuesday, 15th, 11am Where: AA3195 Abstract: Numerous variable selection methods rely on a two-stage procedure, where a sparsity-inducing penalty is used in the first stage to predict the support, which is then conveyed to the second stage for estimation or inference purposes. In this framework, the first stage screens variables to find a set of possibly relevant variables and the second stage operates on this set of candidate variables, to improve estimation accuracy or to assess the uncertainty associated to the selection of variables. We advocate that more information can be conveyed from the first stage to the second one: we use the magnitude of the coefficients estimated in the first stage to define an adaptive penalty that is applied at the second stage. We give the example of an inference procedure that highly benefits from the proposed transfer of information. The procedure is precisely analyzed in a simple setting, and our large-scale experiments empirically demonstrate that actual benefits can be expected in much more general situations, with sensitivity gains ranging from 50% to 100% compared to state-of-the-art. Keywords: Linear model, Lasso, Variable selection, p-values, False discovery rate, Screen and clean.

1 1

Talk on Monday 14th, 1pm: Natural Gradient-based Algorithms for the Training of Neural Networks
by Jörg Bornschein 14 Déc '15

14 Déc '15

Hi everyone, shortly after NIPS, on Monday, we'll have a talk by Luigi Malagò, Professor at Shinshu University. He will talk about Natural Gradient-based Algorithms for Training of Neural Networks. hope you are all enjoying the conference and see you soon :) j -- Who: Luigi Malagò Title: Natural Gradient-based Algorithms for the Training of Neural Networks When: Monday, 14th December from 1 to 2 pm. Where: AA3195 Abstract: Stochastic gradient descent is a well-known method for the training of the weights of a neural network. Since the minimization of the empirical loss corresponds to an optimization problem defined over a statistical model, the direction of steepest descent is given by the natural gradient, i.e., the Riemannian gradient over a statistical manifold evaluated with respect to the Fisher information metric. However, in the general case, the natural gradient requires the evaluation of the inverse Fisher information matrix, which can be computationally unfeasible for large networks. Different approaches to overcome this issue have been proposed in the literature. In the first part of the talk we introduce natural gradient in the context of manifold optimization, next we review different training algorithms based on natural gradient, which have been proposed in the literature of neural networks. Finally, in the last part of the presentation, we describe different approaches to the efficient computation of the natural gradient, which are used in stochastic optimization. Natural gradient methods for the optimization of the stochastic relaxation of a function, in particular in the high-dimensional setting, could inspire the design of novel strategies for the efficient training of large neural networks.

1 2

Talk by Guido Montufar: Thu. 3.12, 11:00am
by Jörg Bornschein 03 Déc '15

03 Déc '15

Hi, this Thursday, shortly before the IBM visit and before NIPS, we will have a talk by Guido Montufar from the Max Planck Institute for Mathematics in the Sciences in Leipzig: A Theory of Cheap Control in Embodied Systems hope to see many of you there, j -- Abstract & Info -- Who: Guido F. Montufar When: Thursday, December 3rd, 11:00 am Where: AA 3195 A Theory of Cheap Control in Embodied Systems http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.10044… Given a body and an environment, what is the brain complexity needed in order to generate a desired set of behaviors? In this talk I will discuss an approach that links the physical and behavioral constraints of an agent to the required controller complexity. I will exemplify the approach with a conditional restricted Boltzmann machine as a controller architecture and present experiments conducted with a virtual six-legged walking creature.

1 1

TeaTalk this Friday, 2:30pm: Equilibrated adaptive learning rates for non-convex optimization Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us desi
by Jörg Bornschein 27 Nov '15

27 Nov '15

Hi, I would like to announce this weeks Tea Talk: Harm de Vries will talk about "Equilibrated adaptive learning rates for non-convex optimization" and about optimization challenges in deep learning in general. When: Friday, November 27th, 14:30 to 15:30 Where: AA3195 Who: Harm de Vries Title: Equilibrated adaptive learning rates for non-convex optimization Link: http://arxiv.org/abs/1502.04390 == Abstract == Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the so- called equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD per- forms as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent. Hope to see you on Friday, j

1 1

(no subject)
by Jörg Bornschein 12 Nov '15

12 Nov '15

Hi, I would like to announce my own tea-talk for tomorrow -- starting at 2:30pm in AA3195. I will present some joint work with Asja and Samira: Time: 2:30pm - 3:30pm Where: AA3195 Title: Training deep generative (bidirectional) Helmholtz-Machines == Abstract == Unsupervised training of deep generative models containing latent variables and performing inference remains a challenging problem. Various methods have been proposed and many of them train an auxiliary model to perform approximate inference for the generative model which is fitted to the training data. The top-down generative model is typically a directed model that starts from some prior over latent variables at the top, down to a distribution over the observed variables at the bottom. The approximate inference model runs in the opposite direction and is typically trained to efficiently infer high probability latent states given some observed data. After presenting some of the well-known / state-of-the-art approaches (Wake sleep, NVIL and VAE) I will to talk about a new method we have been investigating, called a bidirectional Helmholtz machine (BiHM), that is based on the idea that the generative model should be close to the class of distributions that can be modeled by our approximate inference distribution. We achieve this by interpreting both the top-down and the bottom-up directed models as approximate inference distributions and by defining the target distribution we fit to the training data to be the geometric mean of these two. We present an upper-bound for the log-likelihood of this model and we show that optimizing this bound will pressure the model to stay close to the approximate inference distributions. Hope to see you tomorrow! j

2 2

Fwd: COLLOQUE DES SCIENCES MATHÉMATIQUES DU QUÉBEC (30/10/2015, Emmanuel Candès)
by Yoshua Bengio 29 Oct '15

29 Oct '15

---------- Forwarded message ---------- From: CRM <crm(a)crm.umontreal.ca> Date: Mon, Oct 26, 2015 at 2:36 PM Subject: COLLOQUE DES SCIENCES MATHÉMATIQUES DU QUÉBEC (30/10/2015, Emmanuel Candès) To: activites(a)crm.umontreal.ca ****************************************************************** COLLOQUE DES SCIENCES MATHÉMATIQUES DU QUÉBEC - Montréal http://www.crm.umontreal.ca/Colloques/index.html ****************************************************************** DATE : Le vendredi 30 octobre 2015 / Friday, October 30, 2015 HEURE / TIME : 16 h / 4:00 p.m. CONFERENCIER(S) / SPEAKER(S) : Emmanuel Candès (Stanford University) TITRE / TITLE : A knockoff filter for controlling the false discovery rate LIEU / PLACE : CRM, UdeM, Pav. André-Aisenstadt, 2920, ch. de la Tour, salle 6254 RESUME / ABSTRACT : The big data era has created a new scientific paradigm: collect data first, ask questions later. Imagine that we observe a response variable together with a large number of potential explanatory variables, and would like to be able to discover which variables are truly associated with the response. At the same time, we need to know that the false discovery rate (FDR)---the expected fraction of false discoveries among all discoveries---is not too high, in order to assure the scientist that most of the discoveries are indeed true and replicable. We introduce the knockoff filter, a new variable selection procedure controlling the FDR in the statistical linear model whenever there are at least as many observations as variables. This method works by constructing fake variables, knockoffs, which can then be used as controls for the true variables; the method achieves exact FDR control in finite sample settings no matter the design or covariates, the number of variables in the model, and the amplitudes of the unknown regression coefficients, and does not require any knowledge of the noise level. This is joint work with Rina Foygel Barber. ****************************************************************** Responsable(s) : Yvan Saint-Aubin (yvan.saint-aubin(a)umontreal.ca) Iosif Polterovich (iosif.polterovich(a)umontreal.ca) Henri Darmon (darmon(a)math.mcgill.ca) David A. Stephens (dstephens(a)math.mcgill.ca)

1 0

Tea Talk by Sarath Chandar on Friday (30th, 2:30pm)
by Jörg Bornschein 27 Oct '15

27 Oct '15

Hi everyone, this Friday at 2:30pm we will have a talk by Sarath Chandar about Correlational Neural Networks. Looking forward to the talk and hope to see you there, Jorg == Details & Abstract == Title: Correlational Neural Networks Speaker: Sarath Chandar Time: Friday, October 30th, 2:30pm Location: AA3195 Abstract: Common Representation Learning (CRL), wherein different descriptions (or views) of the data are embedded in a common subspace, is receiving a lot of attention recently. Two popular paradigms here are Canonical Correlation Analysis (CCA) based approaches and Autoencoder (AE) based approaches. CCA based approaches learn a joint representation by maximizing correlation of the views when projected to the common subspace. AE based methods learn a common representation by minimizing the error of reconstructing the two views. Each of these approaches has its own advantages and disadvantages. For example, while CCA based approaches outperform AE based approaches for the task of transfer learning, they are not as scalable as the latter. In this work we propose an AE based approach called Correlational Neural Network (CorrNet), that explicitly maximizes correlation among the views when projected to the common subspace. Through a series of experiments, we demonstrate that the proposed CorrNet is better than the above mentioned approaches with respect to its ability to learn correlated common representations. Further, we employ CorrNet for several cross language tasks and show that the representations learned using CorrNet perform better than the ones learned using other state of the art approaches. CorrNet can be easily extended to the case where you have more than 2 views. We demonstrate this by applying CorrNet for two specific downstream applications: cross language document classification across 12 different languages and multilingual multimodal retrieval. Links: 1. Correlational Neural Networks ( http://arxiv.org/abs/1504.07225 ) 2. Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning ( http://arxiv.org/abs/1510.03519 ) ====

1 0

Tea Talk by Galin Georgiev this Friday (23rd, 2:30pm)
by Jörg Bornschein 21 Oct '15

21 Oct '15

Dear all, after a long pause we will have a tea talk this Friday at 2:30pm in AA3195. We will have Galin Georgiev talk about Universal neural nets: Gibbs machines and ACE == Abstract == We study a class of neural nets - Gibbs machines - which are a type of variational auto-encoders, designed for gradual learning. They offer an universal platform for incrementally adding newly learned features, including physical symmetries in space/time. Combining them with classifiers gives rise to a brand of universal generative neural nets - stochastic auto-classifier-encoders (ACE). ACE preserve the non-Gaussian and clustering nature of real-life data and have state-of-the-art performance, both for classification and density estimation for the MNIST data set. Link: http://arxiv.org/abs/1508.06585 Hope to see many of you there, Jorg

1 0

Tea Talk this Wednesday, 1:30pm: Taxi Destination Prediction
by Jörg Bornschein 25 Aoû '15

25 Aoû '15

Dear all, we will have a tea talk by Alexandre de Brébisson, Étienne Simon and Alex Auvolat this Wednesday, 26th August at 1:30pm in AA 3195. They will talk about their successful participation (1st place!) in the Kaggle Taxi Trajectory Prediction challenge: Artificial Neural Networks Applied to Taxi Destination Prediction We describe our first-place solution to the ECML/PKDD discovery challenge on taxi destination prediction. The task consisted in predicting the destination of a taxi based on the beginning of its trajectory, represented as a variable-length sequence of GPS points, and diverse associated meta-information, such as the departure time, the driver id and client information. Contrary to most published competitor approaches, we used an almost fully automated approach based on neural networks and we ranked first out of 381 teams. The architectures we tried use multi-layer perceptrons, bidirectional recurrent neural networks and models inspired from recently introduced memory networks. Our approach could easily be adapted to other applications in which the goal is to predict a fixed-length output from a variable-length sequence. Link to the paper: http://arxiv.org/abs/1508.00021 Hope to see you all there! j

2 1

Fwd: MILA Teatalk about Robotic Priors?
by Yoshua Bengio 11 Aoû '15

11 Aoû '15

---------- Forwarded message ---------- From: Yoshua Bengio <yoshua.bengio(a)gmail.com> Date: Tue, Aug 11, 2015 at 1:59 PM Subject: Fwd: MILA Teatalk about Robotic Priors? To: Lisa Labo <lisa_labo(a)iro.umontreal.ca>, lisa_teatalks(a)iro.umontreal.ca Hi all, We have several visitors following the summer school. In particular Rico Jonschkowski will give a tea-talk this Friday at 2pm, rm 3195, on learning state representations for robotics. -- Yoshua Title: Learning State Representations with Robotic Priors -- And Why Machine Learning Needs Robotics abstract: In this talk, I will present a roboticist's perspective on representation learning. The talk will consist of two parts: In the first part, I will present our work on learning state representations with robotic priors, which are priors about how robots interact with the physical world. The second part will be more philosophical and, I hope, controversial. I will try to extrapolate from the first part and explain why machine learning needs robotics to scale towards AI.

1 0

ICML Practice talk: Thursday June 2nd by Kelvin Xu
by Jörg Bornschein 02 Jul '15

02 Jul '15

Hi everyone, this Thursday Kelvin will present Show, Attend and Tell: Neural Image Caption Generation with Visual Attention which was accepted for this years ICML. Looking forward to see you there and to get your feedback. Location: AA-3195 Time: Thursday, July 2nd, 3:30pm Abstract Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO. Jorg

4 3

Fwd: Soutenance de thèse de Grégoire Mesnil, vendredi 10h00, salle 3195 pavillon André Aisenstadt
by Pascal Vincent 16 Jui '15

16 Jui '15

*[ Apologies for multiple sends. Re-sending, hopefully this time the formatting and accents will be preserved properly ... ]* *Vous êtes tous cordialement invités à assister à la soutenance de thèse de doctorat de Grégoire Mesnil* [ *Note:** To reach a wider audience, the presentation will be given in English. An English abstract of the thesis is given further below *] *Date:* Vendredi 19 juin 2015 *Heure:* 10:00 *Endroit:* salle 3195, pavillon André Aisenstadt 2920 Chemin de la Tour * Titre:* Apprentissage d’Espaces Sémantiques *Résumé:*Dans cette dissertation, nous présentons plusieurs techniques d’apprentissage d’espaces sémantiques pour plusieurs domaines, par exemple des mots et des images, mais aussi à l’intersection de différents domaines. Un espace de représentation est appelé sémantique si des entités jugées similaires par un être humain, ont leur similarité préservée dans cet espace. La première publication présente un enchaînement de méthodes d’apprentissage incluant plusieurs techniques d’apprentissage non supervisé qui nous a permis de remporter la compétition “Unsupervised and Transfer Learning Challenge” en 2011. Le deuxième article présente une manière d’extraire de l’information à partir d’un contexte structuré (177 détecteurs d’objets à différentes positions et échelles). On montrera que l’utilisation de la structure des données combinée à un apprentissage non supervisé permet de réduire la dimensionnalité de 97% tout en améliorant les performances de reconnaissance de scènes de +5% à +11% selon l’ensemble de données. Dans le troisième travail, on s’intéresse à la structure apprise par les réseaux de neurones profonds utilisés dans les deux précédentes publications. Plusieurs hypothèses sont présentées et testées expérimentalement montrant que l’espace appris a de meilleures propriétés de mixage (facilitant l’exploration de différentes classes durant le processus d’échantillonnage). Pour la quatrième publication, on s’intéresse à résoudre un problème d’analyse syntaxique et sémantique avec des réseaux de neurones récurrents appris sur des fenêtres de contexte de mots. Dans notre cinquième travail, nous proposons une façon d’effectuer de la recherche d’image ”augmentée” en apprenant un espace sémantique joint où une recherche d’image contenant un objet retournerait aussi des images des parties de l’objet, par exemple une recherche retournant des images de ”voiture” retournerait aussi des images de ”pare-brises”, ”coffres”, ”roues” en plus des images initiales. *Jury:* Président-rapporteur: Jian-Yun Nie Directeur de recherche: Pascal Vincent Codirecteur: Yoshua Bengio Membre du jury: Aaron Courville Examinateur externe: Prof. Nicolas Usunier; Chaire d’excellence CNRS; Université de Technologie de Compiègne, France; et Facebook AI Research Paris. Représentant du Doyen de la Faculté: Simona Maria Brambati, prof. Adj. Départ. de psychologie – FAS - UdeM *[ English ]* *Title:* Learning Semantic Spaces *Abstract:* In this work, we focus on learning semantic spaces for multiple domains, but also at the intersection of different domains. The semantic space is where the learned representation lives. This space is called semantic if similar entities from a human perspective have their similarity preserved in this space. We use different machine learning algorithms to learn representations with interesting intrinsic properties. The first article presents a pipeline including many different unsupervised learning techniques used to win the Unsupervised and Transfer Learning Challenge in 2011. In the second article, we present a pipeline taking advantage of the structure of the data for a scene classification problem. This approach allows us to drastically reduce the dimensionality while improving significantly on the scene recognition accuracy. The third article focuses on the space structure learned by deep representations. We show that performing the sampling procedure from deeper levels of representation space explores more of the different classes. In the fourth article, we tackle a semantic parsing problem with several Recurrent Neural Network architectures taking as input context windows of word embeddings. In the fifth article, an investigation on learning a single semantic space at the intersection of words and images is presented. We propose a way to perform ”augmented search” where a search on an image containing an object would also return images of the object’s parts.

1 0

Fwd: Colloque du DIRO ce *vendredi* : Louis-Philippe Morency
by Kyung Hyun Cho 19 Mai '15

19 Mai '15

FYI: The tea talk this week by L-P has also been announced as a part of the Department's Colloquia. ---------- Forwarded message ---------- From: Gilles Brassard <brassard(a)iro.umontreal.ca> Date: Tue, May 19, 2015 at 5:26 AM Subject: Colloque du DIRO ce *vendredi* : Louis-Philippe Morency To: seminaires(a)iro.umontreal.ca Cc: Kyung Hyun Cho <cho.k.hyun(a)gmail.com> Bonjour à tous Le Colloque du DIRO reprend du service ce vendredi 22 mai à 14h00 au AA3195. Veuillez noter le jour et l'heure inhabituels. Conférencier : Louis-Philippe Morency Affiliation : Carnegie Mellon University Invité de : Yoshua Bengio Titre : Multimodal Machine Learning: Modeling Human Communication Dynamics Résumé : Human face-to-face communication is a little like a dance, in that participants continuously adjust their behaviors based on verbal and nonverbal cues from the social context. Today's computers and interactive devices are still lacking many of these human-like abilities to hold fluid and natural interactions. Leveraging recent advances in machine learning, audio-visual signal processing and computational linguistic, my research focuses on creating human-computer interaction (HCI) technologies able to analyze, recognize and predict human subtle communicative behaviors in social context. I formalize this new research endeavor with a Human Communication Dynamics framework, addressing four key computational challenges: behavioral dynamic, multimodal dynamic, interpersonal dynamic and societal dynamic. Central to this research effort is the introduction of new probabilistic models able to learn the temporal and fine-grained latent dependencies across behaviors, modalities and interlocutors. In this talk, I will present some of our recent achievements modeling multiple aspects of human communication dynamics, motivated by applications in healthcare (depression, PTSD, suicide, autism), education (learning analytics), business (negotiation, interpersonal skills) and social multimedia (opinion mining, social influence). Biographie: Louis-Philippe Morency is Assistant Professor in the Language Technology Institute at the Carnegie Mellon University where he leads the Multimodal Communication and Machine Learning Laboratory (MultiComp Lab). He received his Ph.D. and Master degrees from MIT Computer Science and Artificial Intelligence Laboratory. In 2008, Dr. Morency was selected as one of "AI's 10 to Watch" by IEEE Intelligent Systems. He has received 7 best paper awards in multiple ACM- and IEEE-sponsored conferences for his work on context-based gesture recognition, multimodal probabilistic fusion and computational models of human communication dynamics. For the past three years, Dr. Morency has been leading a DARPA-funded multi-institution effort called SimSensei which was recently named one of the year’s top ten most promising digital initiatives by the NetExplo Forum, in partnership with UNESCO. Venez nombreux... quant à moi, je suis/serai à Vienne ! - Gilles Brassard

1 0

Talk by Dumitru Erhan (Google) on Monday (27 April) at 14.00
by Kyung Hyun Cho 27 Avr '15

27 Avr '15

Dear all, Our proud alumni (alumnus?) Dumitru Erhan at Google will visit the lab on Monday and give a talk on how Google tackles the task of image understanding. The talk will be in the usual place AA3195 starting at 14.00. Hope to see many of you there! - Cho === Speaker: Dr. Dumitru Erhan, Google Date/Time: 27 April 2015 @14.00 Place: AA3195 Title: Image Understanding at Google Abstract: Extracting knowledge from image data is a difficult challenge, but recent advances in deep conv-nets have been quite the enabler and have made incredible impact inside Google. I'll give a whirl-wind tour of our efforts very large-scale design and training of networks for categorization, localization, scene understanding, compact embedding learning, image captioning and more speculative thoughts on image-based question & answering.

2 1