Hi all,
This Wednesday, same place AA3195, same time 13:00, we have a double tea talk featured. First David will present his (and co-authors) paper accepted at ICLR 2014. After that I will do a practice talk for my oral at ICLR as well.
Be warned that this tea-talk might take longer than 1h all together.
Talk by: David Warde Farley
Title :An empirical analysis of dropout in piecewise linear networks
Abstract:
The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient.
paper on openreviewhttp://openreview.net/document/f4c625c6-b0eb-4fd3-ab50-25182fe68733#f4c625c6-b0eb-4fd3-ab50-25182fe68733
Talk by: Razvan Pascanu (practice talk of 15 +5 minutes oral presentation)
Tile : Revisiting natural gradient for deep networks
Abstract:
The aim of this paper is three-fold. First we show that Hessian-Free (Martens, 2010) and Krylov Subspace Descent (Vinyals and Povey, 2012) can be described as implementations of natural gradient descent due to their use of the extended Gauss-Newton approximation of the Hessian. Secondly we re-derive natural gradient from basic principles, contrasting the difference between two versions of the algorithm found in the neural network literature, as well as highlighting a few differences between natural gradient and typical second order methods. Lastly we show empirically that natural gradient can be robust to overfitting and particularly it can be robust to the order in which the training data is presented to the model.
paper on openreviewhttp://openreview.net/document/1cd7651c-8029-457e-ae24-5fbca0f3a6a7#1cd7651c-8029-457e-ae24-5fbca0f3a6a7
I hope to see many of you there, Best,
Razvan