Lisa_seminaires Septembre 2017

lisa_seminaires@iro.umontreal.ca

4 participants
6 discussions

[Tea Talk] Ioannis Mitliagkas, Fri Sep 29, 10:30AM, AA6214
by Michael Noukhovitch 29 Sep '17

29 Sep '17

This week we have a cool new MILA professor, *Ioannis Mitliagkas,* giving a talk this *Friday Sep 29* at a new, earlier time: *10:30AM* in room *AA6214*. If time permits, he'll also be giving details of his winter course on topics in AI! Hope to see you all there! Don't forget about the earlier time!! - Michael *KEYWORDS* optimization, YellowFin, large scale DL, GAN stabilization *TITLE* Understanding momentum dynamics for faster training, better scaling, and easier tuning *ABSTRACT* This talk revolves around Polyak’s momentum gradient descent method, also known as ‘momentum’. Its stochastic version, momentum stochastic gradient descent (SGD), is one of the most commonly used optimization methods in deep learning. Throughout the talk we will study a number of important properties of this versatile method, and see how this understanding can be used to engineer better deep learning systems. I will first go over the basic formulation of momentum. Then I will summarize a theoretical result on a previously unknown connection between momentum dynamics and asynchronous optimization. Understanding this connection, allows us to improve the efficiency of large-scale deep learning systems. I will go over a recent collaboration with Intel and the National Energy Research Scientific Computing Center (NERSC) on a 15 PetaFLOP system consisting of 9600 nodes. Finally, I will demonstrate how analyzing the behavior of momentum on simple objectives can lead to tuning rules for its learning rate and momentum hyperparameters. Our implementation of these rules is called YellowFin and is a simple adaptive method that can handle different objectives, as well as varying asynchronous dynamics, without hand-tuning. Yellowfin often outperforms state-of-the-art adaptive methods. At the end of the talk, I will discuss some preliminary thoughts on the training dynamics of GANs and some ideas on how momentum dynamics can, again, play a key role in stabilizing adversarial training. Finally, if time permits, I will give an outline my other research interests, as well as a rough plan for my upcoming ’Topics in AI’ class. *BIO* Ioannis Mitliagkas is an assistant professor in the Department of Computer Science and Operations Research (DIRO) at the University of Montreal. Before that, he was a Postdoctoral Scholar with the Department of Statistics and Computer Science at Stanford University. He obtained his Ph.D. from the Department of Electrical and Computer Engineering at the University of Texas at Austin. His research focuses on statistical learning and inference problems, with work in efficient large-scale and distributed algorithms, theoretical and data-dependent guarantees and tuning complex systems. His recent work includes understanding and optimizing the scanning used in Gibbs sampling for inference, as well as understanding the interaction between optimization and the dynamics of large-scale learning systems.

2 1

Anirudh Goyal, September 15, AA6214, 11AM
by Michael Noukhovitch 15 Sep '17

15 Sep '17

Hi All! Dima is going on an internship so I'll be organizing tea talks for the next couple of months! This week, our very own *Anirudh* will present his NIPS paper (congrats btw!) it will be at *AA6214 on Friday September 15th, 11AM *Hope to see everyone there! *Title:* Z-Forcing: Training Stochastic Recurrent Networks *Abstract:* Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We propose a novel sequential latent variable model unifying successful ideas from recently proposed architectures: each step in the sequence is associated with a latent variable that is used to condition the recurrent dynamics for future steps. Training is performed with amortised variational inference where the approximate posterior is augmented with a RNN that runs backward through the sequence. In addition to maximizing the variational lower bound, we ease training of the latent variables by adding an auxiliary cost which forces them to reconstruct the state of the backward recurrent network. This provides the latent variables with a task-independent objective that enhances the performance of the overall model. Although being conceptually simple, our model achieves state-of-the-art results on standard speech benchmarks such as TIMIT and BLIZZARD and competitive performance on sequential MNIST. Finally, we apply our model on language modeling in the IMDB dataset. The auxiliary cost is crucial for learning interpretable latent variables. Michael

2 2

[TeaTalk] Karan Grewal, September 8, 13:45, AA6214
by Dzmitry Bahdanau 08 Sep '17

08 Sep '17

Hi all, Our next speaker is Karan Grewal, who is currently an intern at MILA. The talk will take place at *AA6214, at 13:45, September 8*. Hope to see many of you there! *Title: *Variance Regularizing Adversarial Learning *Abstract:* Generative Adversarial Networks have become a breakthrough towards generating synthetic data in recent years, however many problems may arise during training which result in the generator being unable to learn the desired data distribution/manifold. Most notably, the discriminator often overpowers the generator, which leads to a poor training signal and vanishing gradients. We study the effects of the discriminator's unnormalized output distribution on the generator's ability to learn. To combat this problem, we propose Variance Regularizing Adversarial Learning -- regularizing the discriminator's unnormalized output distribution to fit a mixture of gaussians, yielding higher variance and hence a lipschitz-like discriminator function. To do this, we provide two methods: (1) using the KL-divergence as a penalty in the discriminator's loss and (2) playing meta-adversarial games to force the discriminator to fit a MoG. We show that our models are robust in the presence of high training ratios and compare them to other variants of GANs. *Bio:* Karan Grewal is an intern at MILA and a senior undergraduate student at the University of Toronto. He is working on improving and stabilizing generative models with Devon Hjelm and Yoshua Bengio. Previously, he worked on applying machine learning to social contexts. Dima

1 3

predoctoral thesis proposal, Taesup Kim, Tuesday 5 Sept, 15h30
by Yoshua Bengio 03 Sep '17

03 Sep '17

Hello, Taesup Kim, MILA PhD candidate, will present his predoctoral subject proposal Tuesday September 5 at 15h30 (room 3395, PAA): *Title: Dynamic Neural Networks: Learning to Adapt to DataAbstractRecent advances in deep neural networks have shown significant results on several machine learning and pattern recognition problems. While these achievements are impressive that some are already working on real-world products, there still exists a lot of potential innovations inspired by human behaviors and we cover in this proposal some related approaches to imitate human using deep neural networks involving both generative and discriminative models. Especially, we introduce some dynamic neural networks having dynamic modules that operate differently and adaptively over different data to improve both the optimization process and the performance. Related future works are also presented to show how the dynamic neural networks can be applied broadly over different domains and applications.* Please come and learn about his research and support him! *-- Yoshua*

1 0

predoctoral thesis proposal, Chinnadhurai Sankar, Tuesday 5 Sept, 13h30
by Yoshua Bengio 03 Sep '17

03 Sep '17

Hello, Chinnadhurai Sankar, MILA PhD candidate, will present his predoctoral subject proposal Tuesday September 5 at 13h30 (room 3395, PAA): *Title: Problems in Non-Goal Oriented Dialog Systems* *Abstract:* *This predoctoral presentation will mainly focus on data-driven approaches to non-goal oriented dialog systems research. I shall talk about two dialog systems projects that I have been part of. First, I shall propose a reinforcement learning based response selection model for dialogue management. The proposed method was implemented & tested in the Amazon Alexa Competition and shown to demonstrate superior performance when compared to traditional heuristics based approaches and supervised learning based approaches. **Next, I shall introduce a hybrid retrieval based generative dialog system architecture which is designed to provide a sense of specificity in the generated responses.* Please come and learn about his research and support him! *-- Yoshua*

1 0

[TeaTalk] Iulian Serban (Wednesday, Aug 30) and Andrew Jesson (Friday, Sept 1)
by Dzmitry Bahdanau 01 Sep '17

01 Sep '17

Hi all, We are lucky to have two speakers this week! First,* Iulian Serban* will tell us about the progress of MILA team in Alexa challenge (*Z-209 between 14h00 - 15h00, August 30th*). Next, we will learn about deep learning with extreme class imbalance from *Andrew Jesson *from Imagia (*AA6214, 13:45 - 14:45, September 1*). Please find the details below. Dima *Speaker:* Iulian Serban *Title:* A Deep Reinforcement Learning Chatbot *Abstract*: We present MILABOT: a deep reinforcement learning chatbot developed at MILA for the Amazon Alexa Prize competition. MILABOT is capable of conversing with humans on popular small talk topics through both speech and text. The system consists of an ensemble of natural language generation and retrieval models, including sequence-to-sequence and latent variable neural network models. By applying reinforcement learning to crowd-sourced data and real-world user interactions, the system has been trained to select an appropriate response from the models in its ensemble. The system has been evaluated through A/B testing with real-world users, where it performed excellent compared to competing systems. *Bio:* Iulian Serban is a PhD candidate at MILA, where he investigates deep learning and reinforcement learning methods for dialogue and other natural language processing applications. Iulian is supervised by Yoshua Bengio, Aaron Courville and Joelle Pineau. He holds an MSc in machine learning from University College London, and a BSc in theoretical mathematics and statistics from Copenhagen University. Previously, Iulian worked at DigiCorpus ApS, where he developed a physiotherapy computer vision system, now deployed in clinics across Denmark. *Speaker:* Andrew Jesson *Title:* CASED: Curriculum Adaptive Sampling for Extreme Data Imbalance *Abstract: *We introduce CASED, a novel curriculum sampling algorithm that facilitates the optimization of deep learning segmentation or detection models on data sets with extreme class imbalance. We evaluate the CASED learning framework on the task of lung nodule detection in chest CT. In contrast to two-stage solutions wherein nodule candidates are first proposed by a segmentation model and then refined by a second detection stage, CASED improves the training of deep nodule segmentation models (e.g. UNet) to the point where state of the art results are achieved using only a trivial detection stage. CASED improves the optimization of deep segmentation models by allowing them to first learn how to distinguish nodules from their immediate surroundings, while continuously adding a greater proportion of difficult-to-classify global context, until uniformly sampling from the empirical data distribution. Using CASED during training yields a minimalist proposal to the lung nodule detection problem competitive the LUNA16 nodule detection benchmark with an average sensitivity score of 88.7%. Furthermore, we find that models trained using CASED are robust to nodule annotation quality by showing that comparable results can be achieved when only a point and radius for each ground truth nodule are provided during training. Finally, the CASED learning framework makes no assumptions with regard to imaging modality or segmentation target and so should generalize to other medical imaging problems where class imbalance is a persistent problem. *Bio:* Andrew Jesson is the leader of the new A.I. Research team at Imagia. He is currently subitting his Master's thesis in electrical engineering with a focus on medical imaging at McGill University. Andrew has spent the last 12 years in Montreal where he has aslo earned a B.A. in philosophy from Concordia University and a B.Eng from McGill.

2 6

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Lisa_seminaires Septembre 2017