Hi,
Tomorrow we will be treated to a talk by Pascal Vincent on his recent work on the theoretical underpinning of Denoising Autoencoders. Hope to see you there.
Time &Date: Thursday Nov. 25th, 14h30 Location: Lisa Lab (AA3256)
TITLE: A connection between Score Matching and Denoising Autoencoders
ABSTRACT:
In this tea talk I will present my recent findings on a connection between two apparently very different techniques: denoising autoencoders and score matching.
Denoising autoencoders have been previously shown to be competitive al- ternatives to Restricted Boltzmann Machines for unsupervised pre-training of each layer of a deep architecture. I will show that a simple denoising au- toencoder training criterion is equivalent to matching the score of a specific energy based model to that of a non-parametric Parzen density estimator of the data. This yields several useful insights. It defines a proper probabilistic model for the denoising autoencoder technique which makes it in principle possible to sample from them or to rank examples by their energy. It suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives. It justifies the use of tied weights between the encoder and decoder, and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models.
Cheers, Aaron
Afficher les réponses par date