Correction.

There is a good reason why it was not done before. When I was working on conv nets with Yann LeCun and Patrice Simard in the early 90's some people had tried it but there was no gain. The main reason is that we only had very few channels (like 1 in the input and 5 in the first layer) in the lower layers (where most of the computation took place). When the number of channels becomes large, the advantage of doing the FFT greatly increases because the log(n) overhead can be compensated by the NxN re-use of it through all the NxN channel combinations (N channels in the previous layer times N channels in the next). Also, I don't remember that somebody thought about the advantage brought by this re-use, but I was not involved in this directly so I am not sure.

-- Yoshua



On Tue, May 27, 2014 at 1:49 PM, Guillaume Alain <guillaume.alain.umontreal@gmail.com> wrote:
*spoiler alert* 

Ian is correct in pointing this out, but that's the main point of my talk.

The FFT is not new at all. Everybody knew about it.

It's just that nobody felt like revisiting the idea by actually implementing it and taking measurements.


On Tue, May 27, 2014 at 1:46 PM, Ian Goodfellow <goodfellow.ian@gmail.com> wrote:
FFTs have been used to speed up convolution since at least the 1960s. I'm not sure why this isn't done more in the neural nets community, but this year's ICLR paper is not the place where the idea was first proposed.


2014-05-27 13:21 GMT-04:00 Kyung Hyun Cho <cho.k.hyun@gmail.com>:
Dear all,

Guillaume Alain will tell us about the recently proposed technique of using FFT for fast convolutional operation tomorrow. See below for the detail.

See you there!
- Cho

============

- Speaker: Guillaume Alain
- Date and Time: 28 May 2014 @ 13.00
- Place: AA3195
- Paper to be Discussed: Michael Mathieu, Mikael Henaff, Yann LeCun. Fast Training of Convolutional Networks through FFTs. ICLR 2014

_______________________________________________
Lisa_labo mailing list
Lisa_labo@iro.umontreal.ca
https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo



_______________________________________________
Lisa_labo mailing list
Lisa_labo@iro.umontreal.ca
https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo



_______________________________________________
Lisa_labo mailing list
Lisa_labo@iro.umontreal.ca
https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo