[Tea Talk] Nicolas Le Roux (Google) May 25 2018 10:30 AM AA3195
This week we will have two tea talks (morning and afternoon) on Friday! Wow! The first talk will be *Nicolas Le* *Roux * on *May 25 2018* at *10:30 AM* in room *AA3195* (note the room change, we'll most likely be in 3195 for the summer). Streaming is, as always, at this link: https://bluejeans.com/809027115/webrtc but watch it live because there will be no recordings! Don't waver on going to this talk, it is sure to be invariantly excellent! Michael *TITLE* An exploration of variance reduction techniques in stochastic optimization *KEYWORDS *deep learning theory, optimization *ABSTRACT* I will present recent and ongoing work on reducing the variance in stochastic optimization techniques to speed-up and simplify the resulting algorithms. In particular, stochastic gradient methods can suffer from high variance, limiting their convergence speed. While variance reduction techniques exist in the finite case, they are rarer in the online case. We demonstrate how an increasing momentum offers variance reduction in the online case, at the expense of bias, and how that bias can be countered by an extrapolation step. The resulting algorithm differs from iterate averaging in only a factor, but, in the context of the minimization of a quadratic function, this difference is enough to lead to the first algorithm converging both linearly in the noiseless and sublinearly in the homoscedastic noise case when using a constant stepsize. *BIO* *Nicolas Le Roux got an MSc in Applied Maths from Ecole Centrale Paris and an MSc in Maths, Learning and Vision from ENS Cachan. He got his PhD in 2008 from University of Montreal where he worked with Yoshua Bengio on neural networks in general and their optimisation in particular. He then moved to Microsoft Research Cambridge to work on generative models of images with John Winn. In 2010, he joined Inria in Francis Bach's team to work on large-scale convex optimisation. From 2012 to 2017, he created and managed the research team at Criteo in Paris. He joined Google Brain Montreal in 2017 where he now works on large-scale optimization and reinforcement learning.*
Afficher les réponses par date
For those of you who attended my talk at the Workshop on learning theory, this one will fill in some of the gaps I mentioned. On Tue, 22 May 2018 at 12:17, Michael Noukhovitch <mnoukhov@gmail.com> wrote:
This week we will have two tea talks (morning and afternoon) on Friday! Wow!
The first talk will be *Nicolas Le* *Roux * on *May 25 2018* at *10:30 AM* in room *AA3195* (note the room change, we'll most likely be in 3195 for the summer).
Streaming is, as always, at this link: https://bluejeans.com/809027115/webrtc but watch it live because there will be no recordings!
Don't waver on going to this talk, it is sure to be invariantly excellent! Michael
*TITLE* An exploration of variance reduction techniques in stochastic optimization
*KEYWORDS *deep learning theory, optimization
*ABSTRACT* I will present recent and ongoing work on reducing the variance in stochastic optimization techniques to speed-up and simplify the resulting algorithms. In particular, stochastic gradient methods can suffer from high variance, limiting their convergence speed. While variance reduction techniques exist in the finite case, they are rarer in the online case. We demonstrate how an increasing momentum offers variance reduction in the online case, at the expense of bias, and how that bias can be countered by an extrapolation step. The resulting algorithm differs from iterate averaging in only a factor, but, in the context of the minimization of a quadratic function, this difference is enough to lead to the first algorithm converging both linearly in the noiseless and sublinearly in the homoscedastic noise case when using a constant stepsize.
*BIO* *Nicolas Le Roux got an MSc in Applied Maths from Ecole Centrale Paris and an MSc in Maths, Learning and Vision from ENS Cachan. He got his PhD in 2008 from University of Montreal where he worked with Yoshua Bengio on neural networks in general and their optimisation in particular. He then moved to Microsoft Research Cambridge to work on generative models of images with John Winn. In 2010, he joined Inria in Francis Bach's team to work on large-scale convex optimisation. From 2012 to 2017, he created and managed the research team at Criteo in Paris. He joined Google Brain Montreal in 2017 where he now works on large-scale optimization and reinforcement learning.*
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... <https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... <https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> . _______________________________________________ Lisa_teatalk mailing list Lisa_teatalk@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_teatalk
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... .
-- Nicolas Le Roux
Also, if you'd like to meet Nicolas for a 1 on 1, sign up for a slot on the Google Calendar Appointment: https://calendar.google.com/calendar/selfsched?sstoken=UUUxb01teGhLckE3fGRlZ... Sorry for the extra emails, Michael On Tue, May 22, 2018 at 12:45 PM Nicolas Le Roux <nlr@google.com> wrote:
For those of you who attended my talk at the Workshop on learning theory, this one will fill in some of the gaps I mentioned.
On Tue, 22 May 2018 at 12:17, Michael Noukhovitch <mnoukhov@gmail.com> wrote:
This week we will have two tea talks (morning and afternoon) on Friday! Wow!
The first talk will be *Nicolas Le* *Roux * on *May 25 2018* at *10:30 AM* in room *AA3195* (note the room change, we'll most likely be in 3195 for the summer).
Streaming is, as always, at this link: https://bluejeans.com/809027115/webrtc but watch it live because there will be no recordings!
Don't waver on going to this talk, it is sure to be invariantly excellent! Michael
*TITLE* An exploration of variance reduction techniques in stochastic optimization
*KEYWORDS *deep learning theory, optimization
*ABSTRACT* I will present recent and ongoing work on reducing the variance in stochastic optimization techniques to speed-up and simplify the resulting algorithms. In particular, stochastic gradient methods can suffer from high variance, limiting their convergence speed. While variance reduction techniques exist in the finite case, they are rarer in the online case. We demonstrate how an increasing momentum offers variance reduction in the online case, at the expense of bias, and how that bias can be countered by an extrapolation step. The resulting algorithm differs from iterate averaging in only a factor, but, in the context of the minimization of a quadratic function, this difference is enough to lead to the first algorithm converging both linearly in the noiseless and sublinearly in the homoscedastic noise case when using a constant stepsize.
*BIO* *Nicolas Le Roux got an MSc in Applied Maths from Ecole Centrale Paris and an MSc in Maths, Learning and Vision from ENS Cachan. He got his PhD in 2008 from University of Montreal where he worked with Yoshua Bengio on neural networks in general and their optimisation in particular. He then moved to Microsoft Research Cambridge to work on generative models of images with John Winn. In 2010, he joined Inria in Francis Bach's team to work on large-scale convex optimisation. From 2012 to 2017, he created and managed the research team at Criteo in Paris. He joined Google Brain Montreal in 2017 where he now works on large-scale optimization and reinforcement learning.*
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... <https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... <https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> . _______________________________________________ Lisa_teatalk mailing list Lisa_teatalk@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_teatalk
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... .
-- Nicolas Le Roux
Nicolas said he's also free to meet people 4 - 5pm, sign up here: https://calendar.google.com/calendar/selfsched?sstoken=UUUxb01teGhLckE3fGRlZ... Sorry for the repeated emails On Tue, May 22, 2018 at 1:09 PM Michael Noukhovitch <mnoukhov@gmail.com> wrote:
Also, if you'd like to meet Nicolas for a 1 on 1, sign up for a slot on the Google Calendar Appointment: https://calendar.google.com/calendar/selfsched?sstoken=UUUxb01teGhLckE3fGRlZ...
Sorry for the extra emails, Michael
On Tue, May 22, 2018 at 12:45 PM Nicolas Le Roux <nlr@google.com> wrote:
For those of you who attended my talk at the Workshop on learning theory, this one will fill in some of the gaps I mentioned.
On Tue, 22 May 2018 at 12:17, Michael Noukhovitch <mnoukhov@gmail.com> wrote:
This week we will have two tea talks (morning and afternoon) on Friday! Wow!
The first talk will be *Nicolas Le* *Roux * on *May 25 2018* at *10:30 AM* in room *AA3195* (note the room change, we'll most likely be in 3195 for the summer).
Streaming is, as always, at this link: https://bluejeans.com/809027115/webrtc but watch it live because there will be no recordings!
Don't waver on going to this talk, it is sure to be invariantly excellent! Michael
*TITLE* An exploration of variance reduction techniques in stochastic optimization
*KEYWORDS *deep learning theory, optimization
*ABSTRACT* I will present recent and ongoing work on reducing the variance in stochastic optimization techniques to speed-up and simplify the resulting algorithms. In particular, stochastic gradient methods can suffer from high variance, limiting their convergence speed. While variance reduction techniques exist in the finite case, they are rarer in the online case. We demonstrate how an increasing momentum offers variance reduction in the online case, at the expense of bias, and how that bias can be countered by an extrapolation step. The resulting algorithm differs from iterate averaging in only a factor, but, in the context of the minimization of a quadratic function, this difference is enough to lead to the first algorithm converging both linearly in the noiseless and sublinearly in the homoscedastic noise case when using a constant stepsize.
*BIO* *Nicolas Le Roux got an MSc in Applied Maths from Ecole Centrale Paris and an MSc in Maths, Learning and Vision from ENS Cachan. He got his PhD in 2008 from University of Montreal where he worked with Yoshua Bengio on neural networks in general and their optimisation in particular. He then moved to Microsoft Research Cambridge to work on generative models of images with John Winn. In 2010, he joined Inria in Francis Bach's team to work on large-scale convex optimisation. From 2012 to 2017, he created and managed the research team at Criteo in Paris. He joined Google Brain Montreal in 2017 where he now works on large-scale optimization and reinforcement learning.*
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... <https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... <https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> . _______________________________________________ Lisa_teatalk mailing list Lisa_teatalk@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_teatalk
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... .
-- Nicolas Le Roux
Starting in 5! On Tue, May 22, 2018 at 1:09 PM Michael Noukhovitch <mnoukhov@gmail.com> wrote:
Also, if you'd like to meet Nicolas for a 1 on 1, sign up for a slot on the Google Calendar Appointment: https://calendar.google.com/calendar/selfsched?sstoken=UUUxb01teGhLckE3fGRlZ...
Sorry for the extra emails, Michael
On Tue, May 22, 2018 at 12:45 PM Nicolas Le Roux <nlr@google.com> wrote:
For those of you who attended my talk at the Workshop on learning theory, this one will fill in some of the gaps I mentioned.
On Tue, 22 May 2018 at 12:17, Michael Noukhovitch <mnoukhov@gmail.com> wrote:
This week we will have two tea talks (morning and afternoon) on Friday! Wow!
The first talk will be *Nicolas Le* *Roux * on *May 25 2018* at *10:30 AM* in room *AA3195* (note the room change, we'll most likely be in 3195 for the summer).
Streaming is, as always, at this link: https://bluejeans.com/809027115/webrtc but watch it live because there will be no recordings!
Don't waver on going to this talk, it is sure to be invariantly excellent! Michael
*TITLE* An exploration of variance reduction techniques in stochastic optimization
*KEYWORDS *deep learning theory, optimization
*ABSTRACT* I will present recent and ongoing work on reducing the variance in stochastic optimization techniques to speed-up and simplify the resulting algorithms. In particular, stochastic gradient methods can suffer from high variance, limiting their convergence speed. While variance reduction techniques exist in the finite case, they are rarer in the online case. We demonstrate how an increasing momentum offers variance reduction in the online case, at the expense of bias, and how that bias can be countered by an extrapolation step. The resulting algorithm differs from iterate averaging in only a factor, but, in the context of the minimization of a quadratic function, this difference is enough to lead to the first algorithm converging both linearly in the noiseless and sublinearly in the homoscedastic noise case when using a constant stepsize.
*BIO* *Nicolas Le Roux got an MSc in Applied Maths from Ecole Centrale Paris and an MSc in Maths, Learning and Vision from ENS Cachan. He got his PhD in 2008 from University of Montreal where he worked with Yoshua Bengio on neural networks in general and their optimisation in particular. He then moved to Microsoft Research Cambridge to work on generative models of images with John Winn. In 2010, he joined Inria in Francis Bach's team to work on large-scale convex optimisation. From 2012 to 2017, he created and managed the research team at Criteo in Paris. He joined Google Brain Montreal in 2017 where he now works on large-scale optimization and reinforcement learning.*
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... <https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... <https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukWMA_7jfFzebONgVbD3Oh7wa%2Bk5eVxta8KYUVg%40mail.gmail.com?utm_medium=email&utm_source=footer> . _______________________________________________ Lisa_teatalk mailing list Lisa_teatalk@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_teatalk
-- You received this message because you are subscribed to the Google Groups "brain-montreal-office-hours" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-montreal-office-hours+unsubscribe@google.com. To post to this group, send email to brain-montreal-office-hours@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-montreal-office-hours/C... .
-- You received this message because you are subscribed to the Google Groups "brain-mon-fte" group. To unsubscribe from this group and stop receiving emails from it, send an email to brain-mon-fte+unsubscribe@google.com. To post to this group, send email to brain-mon-fte@google.com. To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/brain-mon-fte/CAMVr0Vy4KsyukW... .
-- Nicolas Le Roux
participants (2)
-
Michael Noukhovitch -
Nicolas Le Roux