Lisa_seminaires

Télécharger

lisa_seminaires@iro.umontreal.ca

Avril 2014

1 participants
2 discussions

Tea talk Wednesday 9th April @13:00 AA3195 by David WF and Razvan
by Razvan Pascanu 08 Avr '14

08 Avr '14

Hi all, This Wednesday, same place AA3195, same time 13:00, we have a double tea talk featured. First David will present his (and co-authors) paper accepted at ICLR 2014. After that I will do a practice talk for my oral at ICLR as well. Be warned that this tea-talk might take longer than 1h all together. Talk by: David Warde Farley Title :An empirical analysis of dropout in piecewise linear networks Abstract: The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient. paper on openreview<http://openreview.net/document/f4c625c6-b0eb-4fd3-ab50-25182fe68733#f4c625c…> Talk by: Razvan Pascanu (practice talk of 15 +5 minutes oral presentation) Tile : Revisiting natural gradient for deep networks Abstract: The aim of this paper is three-fold. First we show that Hessian-Free (Martens, 2010) and Krylov Subspace Descent (Vinyals and Povey, 2012) can be described as implementations of natural gradient descent due to their use of the extended Gauss-Newton approximation of the Hessian. Secondly we re-derive natural gradient from basic principles, contrasting the difference between two versions of the algorithm found in the neural network literature, as well as highlighting a few differences between natural gradient and typical second order methods. Lastly we show empirically that natural gradient can be robust to overfitting and particularly it can be robust to the order in which the training data is presented to the model. paper on openreview<http://openreview.net/document/1cd7651c-8029-457e-ae24-5fbca0f3a6a7#1cd7651…> I hope to see many of you there, Best, Razvan

1 1

Tea talk Wednesday 2nd April @13:00 AA3195 by Chris Pal
by Razvan Pascanu 02 Avr '14

02 Avr '14

Hi all, This Wednesday, same place AA3195, same time 13:00, we have the following tea talk scheduled: Talk by : Christopher Pal Title: A New Smooth Approximation for Learning with the Zero One Loss with a Probabilistic Interpretation We examine a new form of smooth approximation to the zero one loss which can be viewed as learning using a reformulation of the classical functional form of logistic regression. Our approach is based on using the posterior mean of a novel generalized Beta-Bernoulli formulation. This leads to a generalized logistic function that approximates the zero one loss, but retains a probabilistic formulation. We show how the underlying functional form has a number of interesting properties. The approach is also easily generalized to kernel logistic regression and beyond. We present experiments in which we learn such models using a combination of gradient descent and coordinate descent using localized grid search so as to escape from local minima. Our experiments are indicating that that this formulation can improve upon the performance of canonical models such as classical logistic regression and support vector machines and is more robust to outliers. This work is with Md. Kamrul Hasan and he may present the second half of the talk. I hope to see many of you there, Best, Razvan

1 1