Mars 2014 - Lisa_seminaires - mailman.iro.umontreal.ca

Tea talk Wednesday 2nd April @13:00 AA3195 by Chris Pal
by Razvan Pascanu 02 Avr '14

02 Avr '14

Hi all, This Wednesday, same place AA3195, same time 13:00, we have the following tea talk scheduled: Talk by : Christopher Pal Title: A New Smooth Approximation for Learning with the Zero One Loss with a Probabilistic Interpretation We examine a new form of smooth approximation to the zero one loss which can be viewed as learning using a reformulation of the classical functional form of logistic regression. Our approach is based on using the posterior mean of a novel generalized Beta-Bernoulli formulation. This leads to a generalized logistic function that approximates the zero one loss, but retains a probabilistic formulation. We show how the underlying functional form has a number of interesting properties. The approach is also easily generalized to kernel logistic regression and beyond. We present experiments in which we learn such models using a combination of gradient descent and coordinate descent using localized grid search so as to escape from local minima. Our experiments are indicating that that this formulation can improve upon the performance of canonical models such as classical logistic regression and support vector machines and is more robust to outliers. This work is with Md. Kamrul Hasan and he may present the second half of the talk. I hope to see many of you there, Best, Razvan

1 1

Tea talk Wednesday 26th @13:00 AA3195 by Mehdi
by Razvan Pascanu 24 Mar '14

24 Mar '14

Hi all, This Wednesday, 26th March, at 13:00 room AA3195 Mehdi Mirza is going to present (Ian's) and his submission to ICLR 2014 which was accepted in the conference track. See title and abstract bellow. Jessica Thompson will also present after Mehdi. She will introduce herself to the lab and talk about her work. I hope to see many of you there, Best, Razvan An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks Authors Ian J Goodfellow, Mehdi Mirza, Xia Da, Aaron Courville, Yoshua Bengio Catastrophic forgetting is a problem faced by many machine learning models and algorithms. When trained on one task, then trained on a second task, many machine learning models "forget" how to perform the first task. This is widely believed to be a serious problem for neural networks. Here, we investigate the extent to which the catastrophic forgetting problem occurs for modern neural networks, comparing both established and recent gradient-based training algorithms and activation functions. We also examine the effect of the relationship between the first task and the second task on catastrophic forgetting. We find that it is always best to train using the dropout algorithm--the dropout algorithm is consistently best at adapting to the new task, remembering the old task, and has the best tradeoff curve between these two extremes. We find that different tasks and relationships between tasks result in very different rankings of activation function performance. This suggests the choice of activation function should always be cross-validated.

1 0

Presentation this Friday 11:00 in AA3195 by Christian Gagné
by Razvan Pascanu 20 Mar '14

20 Mar '14

Hi all, This Friday, 11:00 we have a presentation by Christian Gagné from Universite Laval. The room is AA3195. I hope to see many of you there. Title: Evolutionary Computation and Machine Learning for the Engineering of Intelligent Systems Abstract: Evolutionary computation consists in black-box population-based optimization techniques inspired by natural evolution. These metaheuristics are quite successful at solving problems hard to tackle with conventional methods, for example when dealing with strongly non convex multimodal functions, multiple objectives, or solutions represented a complex data structures (e.g., graphs, trees, symbolic expressions). Although evolutionary computation is often considered as a close sibling to machine learning, it is often badly understood and dismissed by researchers in the field, mostly given its ad hoc nature and the impression it lacks strong theoretical foundations. Such considerations are unfortunate, as many criticisms are not taking into account latest developments of the field, nor considering the use of powerful tools provided by evolutionary computation for tackling some of the difficult optimization problems of machine learning. In my talk, I will present the field of evolutionary computation, with a particular emphasis on how it can be used in the context of machine learning. I will also present some methods of evolutionary computing that are relying on machine learning. This will be illustrated by various works done in my group at Université Laval, where we are applying in synergy machine learning and evolutionary computation techniques for solving complex engineering problems. Presenter: Christian Gagné Professeur agrégé Laboratoire de vision et systèmes numériques Département de génie électrique et de génie informatique Université Laval, Québec (Québec), Canada http://vision.gel.ulaval.ca/~cgagne christian.gagne(a)gel.ulaval.ca

1 1

Tea talk Wednesday 19th @13:00 AA3195 by Roland Memisevic
by Razvan Pascanu 17 Mar '14

17 Mar '14

Hi all, I'm glad to announce our next tea talk this Wednesday (19th March), at the usual place AA3195 at 13:00. Roland is going to talk to us about some of his recent work. See title and abstract below. I hope to see many of you there ! Title: Zero-bias autoencoders Abstract: We recently stumbled upon a hidden unit activation function that makes it very easy to train an autoencoder by simply minimizing reconstruction error, without using any additional regularization. The activation function takes the following form: it is zero if the net input to the neuron is smaller than some threshold and it is linear elsewhere. It seems important for the linear part to be truly linear and not affine, in other words the bias term for the linear part should be zero. I will describe some experiments as well as some theoretical insights that help explain why this activation function works so well. Joint work with Kishore Konda and David Krueger. In ASCII art: / / / / / / | | | --------0---------- hidden unit's net input on the x-axis, output on the y-axis. (this picture may be messed up if your mail-reader is smart about font spacing) Best, Razvan

1 0