Hi all,
I'm glad to announce our next tea talk this Wednesday (19th March), at the usual place AA3195 at 13:00. Roland is going to talk to us about some of his recent work. See title and abstract below. I hope to see many of you there !
Title: Zero-bias autoencoders
Abstract: We recently stumbled upon a hidden unit activation function that makes it very easy to train an autoencoder by simply minimizing reconstruction error, without using any additional regularization. The activation function takes the following form: it is zero if the net input to the neuron is smaller than some threshold and it is linear elsewhere. It seems important for the linear part to be truly linear and not affine, in other words the bias term for the linear part should be zero. I will describe some experiments as well as some theoretical insights that help explain why this activation function works so well. Joint work with Kishore Konda and David Krueger.
In ASCII art:
/ / / / / / | | | --------0----------
hidden unit's net input on the x-axis, output on the y-axis. (this picture may be messed up if your mail-reader is smart about font spacing)
Best, Razvan