[Lisa_teatalk] Tea Talk Tomorrow!

Aaron Courville aaron.courville at gmail.com
Wed Aug 4 14:35:52 EDT 2010


Greeting,

After a week hiatus, we are back with a regularly scheduled tea talk.
This week Xavier Glorot will talk about some of his recent work with
Yoshua. Hope to see you all there.

Date and Time: Thursday August 5th, 15h00
Location: LISA lab (AA3256)

title: Deep Sparse Rectifier Neural Networks

abstract:
For multi-layer neural networks, the rectifier activation function,
f(x)=max(0, x), is more consistent to neuroscience observations than the
sigmoid or the hyperbolic tangent. Firstly, in addition to a L1
regularization on the activations, it creates sparse representations with
exact zeros. Secondly, neurons in the cortex seem to work in a linear
regime. In the context of gradient-based optimization and representation
efficiency, direct mathematical advantages arise from those properties.
However, potential intuitive problems remains: non-differentiability at 0,
hard non-linearity, ill-conditionning, unboundedness. We tested this
activation function, and variants, on several image classification
datasets. We show that networks of rectifying neurons yield significantly
better accuracy than hyperbolic tangent networks.


Cheers,
Aaron

-- 
Aaron C. Courville
Département d’Informatique et
de recherche opérationnelle
Université de Montréal
email:Aaron.Courville at gmail.com


More information about the Lisa_teatalk mailing list