Lisa_seminaires

lisa_seminaires@iro.umontreal.ca

396 discussions

Fwd: [Lisa_labo] Talk 21 Nov Fri @15.00 AA3195 by Dr. John Hershey
by Frédéric Bastien 21 Nov '14

21 Nov '14

I forward this as there is an email issue. Fred ---------- Forwarded message ---------- From: Kyung Hyun Cho <cho.k.hyun(a)gmail.com> Date: Fri, Nov 21, 2014 at 11:38 AM Subject: Re: [Lisa_labo] Talk 21 Nov Fri @15.00 AA3195 by Dr. John Hershey To: lisa_labo Labo <lisa_labo(a)iro.umontreal.ca>, " lisa_teatalk(a)iro.umontreal.ca" <lisa_teatalk(a)iro.umontreal.ca>, lisa_seminaires(a)iro.umontreal.ca Cc: John Hershey <hershey(a)merl.com> Dear all, It's a reminder that we have a talk by Dr. John Hershey *today* at 15.00. Hope to see many of you there! - K On Sun, Nov 16, 2014 at 6:00 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > Dr. John Hershey (Mistubishi Electric Research Labs, US) will tell us > about the connection between model-free approaches (e.g. deep neural > networks) and model-based approaches (e.g. probabilistic graphical models) > and how we can utilize this connection. His talk will start from 15.00 next > Friday (21 Nov) at the usual place AA3195. > > Hope to see many of you there! > - Cho > > === > - Speaker: Dr. John Hershey (Mistubishi Electric Research Labs) > - Date/Time: 15.00 - 16.00, 21 Nov > - Place: AA3195 > - Title: Deep Unfolding: Infusing Deep Architectures with Generative > Model Inference > - Abstract: > > Model-based methods and deep neural networks have both been tremendously > successful paradigms in machine learning. In model-based methods, problem > domain knowledge can be built into the constraints of the model, typically > at the expense of difficulties during inference. In contrast, deterministic > deep neural networks are constructed in such a way that inference is > straightforward, but their architectures are rather generic and it can be > unclear how to incorporate problem domain knowledge. This work aims to > obtain the advantages of both approaches. To do so, we start with a > model-based approach and unfold the iterations of its inference method to > form a layer-wise structure. We then decouple the model parameters across > layers to increase the network's learning capacity. This results in novel > neural-network-like architectures that incorporate our model-based > constraints, but can be trained discriminatively to perform fast and > accurate inference. We show how this framework can be applied to a > non-negative matrix factorization model to obtain a new kind of > non-negative deep neural network, that can be trained using a > multiplicative backpropagation-style update algorithm. We present speech > enhancement experiments showing that our approach is competitive with > conventional neural networks despite using far fewer parameters. > > - Bio: > John Hershey is a researcher at Mistubishi Electric Research Labs (MERL), > in Cambridge, MA since 2010. Prior to that, John spent 5 years as a > researcher at IBM's Watson Research Center in New York, in the Speech > Algorithms and Engines group, and one year as visiting researcher in the > speech group at Microsoft Research, in Redmond, WA. His obtained his > Ph.D. at the University of California, San Diego. He is currently working > on machine learning for signal enhancement and separation, speech > recognition, language processing, and adaptive user interfaces. > > > > > _______________________________________________ Lisa_labo mailing list Lisa_labo(a)iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo

1 0

Thesis Defences: Ian Goodfellow (2 Sep) and Razvan Pascanu (3 Sep)
by Kyung Hyun Cho 25 Aoû '14

25 Aoû '14

Dear all, I'd like announce two thesis defences scheduled next week. - On Tuesday, 2 Sep 2014 from 13.00 @C-3151, Pavillon Lionel-Groulx Ian Goodfellow will defend his thesis. The external examiner is Prof. Yann LeCun of New York University. - On Wednesday, 3 Sep 2014 from 12.30 @AA3195 Razvan Pascanu will defend his thesis. The external examiner is Dr. Ilya Sutskever of Google. I hope to see many of you there! - Cho

1 1

Announcements about Tea Talks
by Kyunghyun Cho 09 Mai '14

09 Mai '14

Hi all, I will be taking over the role of arranging tea talks after Razvan graduates and leaves the lab (Congrats, Razvan!) I have a few announcements to make here. (1) If you want to give a teatalk: Please, feel free to contact me about it. As far as I understand, a topic for a teatalk may include; (a) your own work (b) others' work that may interest the members of the lab (c) some new ideas that others may have interest in I'd especially like to emphasize (b). Often, it's easier and faster to understand the concept of a paper, if someone explains it in 'spoken' words, rather than 'written' ones. This will help all the members keep up with the latest and important topics in the field. Also, if you're hosting a visitor at the lab and she/he's willing to give a talk, do let me so that I can book a time and place in advance. (2) Prof. Alain Tapp: Wednesday 21 June @13.00 AA3195 The title and abstract are not available yet, but I will send an email once more when I have them. (3) Dustin Webb: Friday 23 June @13.00 AA3195 Title: Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs Abstract: An accurate dynamics model of a robot is an important ingredient of many algorithms used to solve robotics problems, including motion planning, control, localization, and mapping. Models derived from ﬁrst principles often contain parameters (e.g. mass, moment of inertia, arm lengths, etc.) for which values are unknown. Those which cannot be easily measured must be estimated from the observed behavior of the robot. A good approach to address this problem is to plan control policies for the robot that elicit maximal amounts of information about the parameters of the system, while still achieving other objectives speciﬁed for the robot. In case of parameters subject to drift, this must be done continuously over the lifetime of the robot if costly re-calibrations are to be avoided. In this paper, we introduce a new method that formulates the parameter estimation problem as a continuous partially-observable Markov decision process (POMDP), which plans control policies that optimally trade-off the effort spent on learning parameters and effort spent on achieving regular robot objectives (exploration vs. exploitation), and allow for online, continual parameter estimation. While POMDPs have, until recently, been mostly of theoretical interest due to their inherent complexity, we build on recent advances that allow continuous, Gaussian POMDPs to be approximately-optimally solved in near-real-time rates. We show that the computed control policies lead to improved convergence of the belief of the parameters compared to system identiﬁcation approaches based on applying random controls. Dustin J. Webb, Kyle L. Crandall and Jur van den Berg. Online Parameter Estimation via Real-Time Replanning of Continuous Gaussian POMDPs. ICRA 2014. http://arl.cs.utah.edu/pubs/ICRA2014-ope.pdf

1 0

Tea talk Wednesday 9th April @13:00 AA3195 by David WF and Razvan
by Razvan Pascanu 08 Avr '14

08 Avr '14

Hi all, This Wednesday, same place AA3195, same time 13:00, we have a double tea talk featured. First David will present his (and co-authors) paper accepted at ICLR 2014. After that I will do a practice talk for my oral at ICLR as well. Be warned that this tea-talk might take longer than 1h all together. Talk by: David Warde Farley Title :An empirical analysis of dropout in piecewise linear networks Abstract: The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient. paper on openreview<http://openreview.net/document/f4c625c6-b0eb-4fd3-ab50-25182fe68733#f4c625c…> Talk by: Razvan Pascanu (practice talk of 15 +5 minutes oral presentation) Tile : Revisiting natural gradient for deep networks Abstract: The aim of this paper is three-fold. First we show that Hessian-Free (Martens, 2010) and Krylov Subspace Descent (Vinyals and Povey, 2012) can be described as implementations of natural gradient descent due to their use of the extended Gauss-Newton approximation of the Hessian. Secondly we re-derive natural gradient from basic principles, contrasting the difference between two versions of the algorithm found in the neural network literature, as well as highlighting a few differences between natural gradient and typical second order methods. Lastly we show empirically that natural gradient can be robust to overfitting and particularly it can be robust to the order in which the training data is presented to the model. paper on openreview<http://openreview.net/document/1cd7651c-8029-457e-ae24-5fbca0f3a6a7#1cd7651…> I hope to see many of you there, Best, Razvan

1 1

Tea talk Wednesday 2nd April @13:00 AA3195 by Chris Pal
by Razvan Pascanu 02 Avr '14

02 Avr '14

Hi all, This Wednesday, same place AA3195, same time 13:00, we have the following tea talk scheduled: Talk by : Christopher Pal Title: A New Smooth Approximation for Learning with the Zero One Loss with a Probabilistic Interpretation We examine a new form of smooth approximation to the zero one loss which can be viewed as learning using a reformulation of the classical functional form of logistic regression. Our approach is based on using the posterior mean of a novel generalized Beta-Bernoulli formulation. This leads to a generalized logistic function that approximates the zero one loss, but retains a probabilistic formulation. We show how the underlying functional form has a number of interesting properties. The approach is also easily generalized to kernel logistic regression and beyond. We present experiments in which we learn such models using a combination of gradient descent and coordinate descent using localized grid search so as to escape from local minima. Our experiments are indicating that that this formulation can improve upon the performance of canonical models such as classical logistic regression and support vector machines and is more robust to outliers. This work is with Md. Kamrul Hasan and he may present the second half of the talk. I hope to see many of you there, Best, Razvan

1 1

Tea talk Wednesday 26th @13:00 AA3195 by Mehdi
by Razvan Pascanu 24 Mar '14

24 Mar '14

Hi all, This Wednesday, 26th March, at 13:00 room AA3195 Mehdi Mirza is going to present (Ian's) and his submission to ICLR 2014 which was accepted in the conference track. See title and abstract bellow. Jessica Thompson will also present after Mehdi. She will introduce herself to the lab and talk about her work. I hope to see many of you there, Best, Razvan An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks Authors Ian J Goodfellow, Mehdi Mirza, Xia Da, Aaron Courville, Yoshua Bengio Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models "forget" how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting. We find that it is always best to train using the dropout algorithm--the dropout algorithm is consistently best at adapting to the new task, remembering the old task, and has the best tradeoff curve between these two extremes. We find that different tasks and relationships between tasks result in very different rankings of activation function performance. This suggests the choice of activation function should always be cross-validated.

1 0

Presentation this Friday 11:00 in AA3195 by Christian Gagné
by Razvan Pascanu 20 Mar '14

20 Mar '14

Hi all, This Friday, 11:00 we have a presentation by Christian Gagné from Universite Laval. The room is AA3195. I hope to see many of you there. Title: Evolutionary Computation and Machine Learning for the Engineering of Intelligent Systems Abstract: Evolutionary computation consists in black-box population-based optimization techniques inspired by natural evolution. These metaheuristics are quite successful at solving problems hard to tackle with conventional methods, for example when dealing with strongly non convex multimodal functions, multiple objectives, or solutions represented a complex data structures (e.g., graphs, trees, symbolic expressions). Although evolutionary computation is often considered as a close sibling to machine learning, it is often badly understood and dismissed by researchers in the field, mostly given its ad hoc nature and the impression it lacks strong theoretical foundations. Such considerations are unfortunate, as many criticisms are not taking into account latest developments of the field, nor considering the use of powerful tools provided by evolutionary computation for tackling some of the difficult optimization problems of machine learning. In my talk, I will present the field of evolutionary computation, with a particular emphasis on how it can be used in the context of machine learning. I will also present some methods of evolutionary computing that are relying on machine learning. This will be illustrated by various works done in my group at Université Laval, where we are applying in synergy machine learning and evolutionary computation techniques for solving complex engineering problems. Presenter: Christian Gagné Professeur agrégé Laboratoire de vision et systèmes numériques Département de génie électrique et de génie informatique Université Laval, Québec (Québec), Canada http://vision.gel.ulaval.ca/~cgagne christian.gagne(a)gel.ulaval.ca

1 1

Tea talk Wednesday 19th @13:00 AA3195 by Roland Memisevic
by Razvan Pascanu 17 Mar '14

17 Mar '14

Hi all, I'm glad to announce our next tea talk this Wednesday (19th March), at the usual place AA3195 at 13:00. Roland is going to talk to us about some of his recent work. See title and abstract below. I hope to see many of you there ! Title: Zero-bias autoencoders Abstract: We recently stumbled upon a hidden unit activation function that makes it very easy to train an autoencoder by simply minimizing reconstruction error, without using any additional regularization. The activation function takes the following form: it is zero if the net input to the neuron is smaller than some threshold and it is linear elsewhere. It seems important for the linear part to be truly linear and not affine, in other words the bias term for the linear part should be zero. I will describe some experiments as well as some theoretical insights that help explain why this activation function works so well. Joint work with Kishore Konda and David Krueger. In ASCII art: / / / / / / | | | --------0---------- hidden unit's net input on the x-axis, output on the y-axis. (this picture may be messed up if your mail-reader is smart about font spacing) Best, Razvan

1 0

Fwd: Predoc Presentation Abstract
by Yoshua Bengio 17 Jan '14

17 Jan '14

Hi all, Caglar Gulcehre will defend his thesis proposal next Tuesday at 1:30pm in room 3195 (unless I am mistaken). Please mark your calendars and come to participate in this important event in the life of a PhD student and learn about Caglar's research. -- Yoshua ---------- Forwarded message ---------- From: Çağlar Gülçehre <ca9lar(a)gmail.com> Date: Fri, Jan 17, 2014 at 5:06 AM Subject: Predoc Presentation Abstract To: Yoshua Bengio <bengioy(a)iro.umontreal.ca> Cc: Alain Tapp <tappa(a)iro.umontreal.ca>, Roland Memisevic < roland.memisevic(a)gmail.com> Dear Professors, Please find the title of and abstract of my predoc presentation below: Title:On Challenges of Learning Abstract Tasks with Deep Learning Abstract: Deep learning algorithms have great successes on different benchmarks in the recent years. Deep learning algorithms rely on combining layers of latent factors into hierarchies. The higher levels of the hiearchy can represent more abstract or higher-level features. Nevertheless it is still a weakly-understood phenomenon that under what conditions the learning dynamics during the training of deep learning algorithms give rise to those higher-level features. Motivated from the cultural learning and effects of intermediate hints learned from other individuals on learning high level abstractions, we investigated the performance of several machine learning algorithms on a binary artificial dataset. As a result of our experiments, we observed that without any prior knowledge all the models including the common deep learning algorithms. An interesting characteristic of the problem is that, it is a combination of two nonlinear sub-tasks. We were able to solve this task by providing intermediate level hints to the architecture and by changing the architecture. Our findings suggest that deep learning algorithms can have difficulty to learn high level abstract tasks due to the inherent optimization issues. We considered a novel activation function for deep neural networks called [image: L_p] norm pooling. The unique characteristic of that activation function is that it is a pooling activation function that is learned as part of the learning process of the norm-pooling. The learning of [image: p] values in [image: L_p] can help to generalize into other types of norm pooling methods. Finally I propose new approaches and problems for the future work, specifically the tasks that involves sequential learning and structured prediction such as natural language processing. The problems addressed in this thesis are mainly a consequence of difficult optimization and might be solved via cognitively inspired algorithms with mathematical foundations in the machine learning literature. Best, -- Caglar GULCEHRE

1 0

Talk by Surya Ganguli, this Friday 17th Jan, 14h00-15h00 AA3195
by Razvan Pascanu 14 Jan '14

14 Jan '14

Hi all, This Friday we have a talk by Surya Ganguli at 14h00 in room 3195 Andre Aisenstadt. Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Networks Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed remains finite: for a special class of initial conditions on the weights, very deep networks incur only a finite delay in learning speed relative to shallow networks. We further show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions. We also discuss application to infant semantic development. I hope to see many of you there ! Best, Razvan

2 1

Aller à la page :

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Lisa_seminaires