[Lisa_teatalk] Talk on Monday 14th, 1pm: Natural Gradient-based Algorithms for the Training of Neural Networks

Jörg Bornschein bornj at iro.umontreal.ca
Wed Dec 9 11:34:17 EST 2015


Hi everyone,

shortly after NIPS, on Monday, we'll have a talk by Luigi Malagò, Professor
at Shinshu University. He will talk about Natural Gradient-based Algorithms
for Training of Neural Networks.


hope you are all enjoying the conference and see you soon :)

   j


--
Who: Luigi Malagò
Title: Natural Gradient-based Algorithms for the Training of Neural Networks
When: Monday, 14th December from 1 to 2 pm.
Where: AA3195

Abstract:

Stochastic gradient descent is a well-known method for the training of
the weights of a neural network. Since the minimization of the empirical
loss
corresponds to an optimization problem defined over a statistical model,
the direction of steepest descent is given by the natural
gradient, i.e., the Riemannian gradient over a statistical manifold
evaluated with
respect to the Fisher information metric. However, in the general case, the
natural
gradient requires the evaluation of the inverse Fisher information
matrix, which can be computationally unfeasible for large networks.
Different approaches to overcome this issue have been proposed in
the literature. In the first part of the talk we introduce natural gradient
in the context of
manifold optimization, next we review different training algorithms based
on natural gradient,
which have been proposed in the literature of neural networks. Finally, in
the last part of the
presentation, we describe different approaches to the efficient computation
of the natural
gradient, which are used in stochastic optimization. Natural gradient
methods for the optimization of the stochastic relaxation of a function, in
particular in the high-dimensional setting, could inspire the design of
novel strategies for the efficient training of large neural networks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://webmail.iro.umontreal.ca/pipermail/lisa_teatalk/attachments/20151209/fab6e914/attachment.html 


More information about the Lisa_teatalk mailing list