Lisa_seminaires

lisa_seminaires@iro.umontreal.ca

396 discussions

Démarrer une nNouvelle discussion

[Tea Talk] Cancelled for ICLR
by Michael Noukhovitch 23 Oct '17

23 Oct '17

Tea talk this Friday is *cancelled* due to the ICLR deadline, good luck to everyone rushing! Michael

1 0

[Tea Talk] Hui Jiang (York University), Fri Oct 13, 10:30AM, AA6214
by Michael Noukhovitch 11 Oct '17

11 Oct '17

This week we have a professor from York University, *Hui Jiang,* giving a talk on *Friday Oct 13* at* 10:30AM* in room *AA6214*. Don't be scared just because it's Friday the 13th in December! Come to the tea talk! Michael *KEYWORDS* NLP, Word Embedding, Language Modelling *TITLE* A New General Deep Learning Approach for Natural Language Processing *ABSTRACT* Word embedding techniques, representing each discrete word as a dense vector in continuous high-dimension space, have achieved successes in many natural language processing (NLP) tasks. However, more NLP tasks rely on modelling variable-length sequences of words, not just isolated words. The conventional approach is to formulate these NLP tasks as sequence labelling problems and apply conditional random fields (CRF), convolutional neural networks (CNN) and recurrent neural networks (RNN). In this talk, I will introduce a new general deep learning approach applicable to almost all NLP tasks, not limited to sequence labelling problems. The proposed method is built upon a simple but theoretically-guaranteed lossless encoding method, namely fixed-size ordinally-forgetting encoding (FOFE), which can almost uniquely encode any variable-length word sequence into fixed-size representation [1]. Next, simple feedforward neural networks are used as universal function approximators to map fixed-size FOFE codes to various NLP targets. This framework is appealing since it is elegant and well-founded in theory and meanwhile fairly easy and fast to train in practice. It is totally data-driven without any feature engineering, and equally applicable to a wide range of NLP tasks. In this talk, I will introduce our recent work to apply this approach to several important NLP tasks, such as word embedding [4], language modelling [1], named entity recognition (NER) and mention detection [2], coreference resolution and text categorization. Experiments have shown that the proposed approach yields strong performance in all examined tasks, including Google 1-billion-word language modelling, TAC KBP EDL contests, Pronoun Disambiguation Problem (PDP) in Winograd Schema Challenge [3]. Finally, I will briefly mention some on-going work to apply this approach to more NLP problems, such as factoid Q/A, word sense disambiguation (WSD), parsing and machine translation. References: [1] S. Zhang, H. Jiang, M. Xu, J. Hou, L. Dai, “The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models,” Proc. of ACL 2015. [2] M. Xu, H. Jiang and S. Watcharawittayakul, “A Local Detection Approach for Named Entity Recognition and Mention Detection,” Proc. of ACL 2017. [3] Q. Liu, H. Jiang, A. Evdokimov, Z. Ling, X. Zhu, S. Wei and Y. Hu, "Cause-Effect Knowledge Acquisition and Neural Association Model for Solving A Set of Winograd Schema Problems," Proc. of IJCAI 2017. [4] J. Sanu, M. Xu, Hui Jiang and Q. Liu, “Word Embeddings based on Fixed-Size Ordinally Forgetting Encoding,” Proc. of EMNLP 2017. *BIO* Hui Jiang received B.Eng. and M.Eng. degrees from University of Science and Technology of China (USTC), China and the Ph.D. degree from the University of Tokyo, Japan, all in electrical engineering. Since 2002, he has been working at Department of Electrical Engineering and Computer Science, York University, Toronto, Canada, initially as an assistant professor, then an associate professor and currently a full professor. His current research interests include machine learning, especially deep learning or neural networks, with its applications to speech and audio processing, natural language processing and computer vision. He served as an associate editor for IEEE Trans. on Audio, Speech and Language Processing (T-ASLP) between 2009-2013, and some technical committees of international conferences. He has recently received the 2016 IEEE Signal Processing Society (SPS) Best Paper Award.

1 0

Talk by David Grangier thursday 15h30 room 1360
by Pascal Vincent 11 Oct '17

11 Oct '17

Hello MILA friends, A reminder that tomorrow thursday at 15:30 we will have a talk by David Grangier on a novel and much faster seq2seq architecture for translation "Neural Machine Translation: Achieving Fast Training and Fast Inference with Gated Convolutions." This is part of the "colloques du DIRO series", so there is a traditional coffee+biscuits half-an-hour before. Reposting the full announcement below ( url: http://diro.umontreal.ca/ departement/colloques/une-nouvelle/news/david-grangier- neural-machine-translation-achie-42551/ ) Conférencier : David Grangier, Facebook AI Research, Menlo Park, CA Titre : Neural Machine Translation: Achieving Fast Training and Fast Inference with Gated Convolutions Travail conjoint avec Michael Auli, Yann Dauphin, Angela Fan, Jonas Ghering et Sergey Edunov Local : André-Aisenstadt 1360 Date et heure : Jeudi 12 octobre, 15h30, café-biscuits à 15h Résumé : Neural architectures for Machine Translation (MT) and related language modeling tasks is an active research field. The first part of our talk introduces several architectural changes to the original work of Bahdanau et al. 2014. We replace non-linearities with our novel gated linear units, recurrent units with convolutions and introduce multi-hop attention to allow more complex attention patterns. These changes improve generalization performance, training efficiency and decoding speed. The second part of our talk analyzes the properties of the distribution predicted by the model, examine how predictions differ from their empirical counterpart and we discuss how this influences beam search. Biographie du conférencier : David Grangier is a research scientist at Facebook AI Research, Menlo Park, CA. David earned his PhD in Machine Learning from Ecole Polytechnique Federale de Lausanne advised by Samy Bengio. He worked at different industrial labs, including NEC Labs America (2008-2011), AT&T Research (2011-2012) and Microsoft Research (2012-2014). Currently, David works on machine learning and its application to natural language processing, he is particularly interested in text generation tasks. http://david.grangier.info/ Pour vous préparer, vous pourriez lire... Convolutional Sequence to Sequence Learning Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin - International Conference on Machine Learning (ICML). 2017. Language Modeling with Gated Convolutional Networks Yann N. Dauphin, Angela Fan, Michael Auli and David Grangier - International Conference on Machine Learning (ICML). 2017. Efficient softmax approximation for GPUs Edouard Grave, Armand Joulin, Moustapha Cisse and David Grangier and Hervé Jegou - International Conference on Machine Learning (ICML). 2017. A Convolutional Encoder Model for Neural Machine Translation Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin - Conference of the Association for Computational Linguistics (ACL). 2017. Neural Generation of Text from Structured Data with Application to the Bibliography Domain Remi Lebret, David Grangier and Michael Auli - Conference on Empirical Methods in Natural Language Processing (EMNLP). 2016. Vocabulary Selection Strategies for Neural Machine Translation Gurvan L'Hostis, David Grangier, Michael Auli - arXiv:1610.00072. 2016. Strategies for Training Large Vocabulary Neural Language Models W. Chen, D. Grangier and M. Auli - Conference of the Association for Computational Linguistics (ACL). 2016.

1 0

[Tea Talk] Clement Farabet (NVIDIA) Mon Oct 9, 2pm, Claire-McNicoll Z-209
by Michael Noukhovitch 09 Oct '17

09 Oct '17

Next week, we have a flurry of tea talks and it all starts with VP of AI infrastructure at NVIDIA, *Clement Farabet* on *Monday October 9th *at *2pm* at *Pavilion Claire McNicoll room Z-209*. There will be a *small social *after the event so you can talk directly with NVIDIA researchers! NVIDIA has also been incredibly kind and will bring *FREE FOOD* as well as hold a *RAFFLE!* I don't know what the raffle is for, but will leave it your imagination ;) This is an awesome event, and hope everyone can make it! If you plan to attend, please sign up on their eventbrite link: https://www.eventbrite.com/e/mila-tea-talk-tickets-38158869205 *KEYWORDS *Infrastructure, Self-driving Cars, Domain Transfer *TITLE* Deep Learning Research at NVIDIA *ABSTRACT* NVIDIA is the leading platform for Deep Learning research and invests in a broad range of research projects within key industries, such as graphics and self-driving cars. In this talk I will discuss a few of our projects in-depth: from leveraging synthetic data to reduce our need on data; ray-tracing for real-time virtual worlds; auto face animation for game designers; and auto image transforms for photo editing. I will also talk about how we apply this research work into a product and operate a fast-moving R&D team to build autonomous vehicles. *BIO* Clement Farabet is VP of AI Infrastructure at NVIDIA. His team is responsible for building NVIDIA's next-generation AI platform, leveraging NVIDIA's hardware to enable a broad range of new applications, ranging from self-driving cars to medical imaging. Clement Farabet received a Master’s Degree in Electrical Engineering with honors from Institut National des Sciences Appliquées (INSA) de Lyon, France in 2008. His Master’s thesis work on reconfigurable hardware for deep neural networks was developed at the Courant Institute of Mathematical Sciences of New York University with Professor Yann LeCun, and led to a patent. He then joined Professor Yann LeCun’s laboratory in 2008, as a research scientist. In 2009, he started collaborating with Yale University’s e-Lab, led by Professor Eugenio Culurciello. This joint work later led to the creation of TeraDeep ( www.teradeep.com). In 2010, he started the PhD program at Université Paris-Est, co-advised by Professors Laurent Najman and Yann LeCun. His thesis focused on real-time image understanding/parsing with deep convolutional networks. The main contributions of his thesis were multi-scale convolutional networks and graph-based techniques for efficient segmentations of class prediction maps. He graduated in 2013, and went on to cofound Madbits, a company that focused on representing, understanding and connecting images. Madbits was acquired by Twitter in 2014. At Twitter, he cofounded Cortex, a team of software engineers, data scientists, and research scientists dedicated to developing state-of-the-art machine learning capabilities to refine and enable new products. He subsequently lead and managed a team called Cortex Core, which focused on building a high-leverage modular machine/deep learning platform to power every aspect of the Twitter product (recommendation systems, search, timeline ranking, etc.). This team focused on (1) developing models of text, images, video, users, and (2) making these models seamlessly importable as components of user-facing ML systems.

1 3

[Tea Talk] Phil Bachman (Maluuba MS), Fri Oct 6, 10:30AM, AA6214
by Michael Noukhovitch 06 Oct '17

06 Oct '17

This week we have a researcher from Maluuba Microsoft, *Phil Bachman,* giving a talk on *Friday Oct 6* at the new, earlier time: *10:30AM* in room *AA6214*. See you there! Michael *KEYWORDS* Active learning, Meta-learning, Bayesian *TITLE* Metalearning and active learning -- like chocolate and peanut butter. *ABSTRACT* This talk expounds on the natural synergy of meta and active learning. The general scheme of metalearning is to swap hard-coded procedures that output models for trainable models (that output models). For example, one can use many related classification problems to train a model which outputs a classifier given an input set of labeled examples. In effect, the trained "set2func" model replaces the role of, e.g., SGD in producing a classifier for each problem. >From a Bayesian perspective, the set2func model can be interpreted as encoding both a prior over classifiers and a procedure for performing inference w.r.t. the learned prior given some labeled data. Existing approaches to meta classification can be interpreted as learning a prior over classifiers, which is co-adapted with some mechanism for performing MAP inference. E.g., the "base" parameters learned by MAML give the mean of a Gaussian prior over parameterizations of the "base" model. Adapting the base parameters to a new problem instance via SGD can be interpreted as approximate posterior inference (see, e.g. "Early Stopping as Non-parametric Variational Inference"). Current metalearning methods perform MAP inference, which can be particularly limiting in the case of tiny sets of labeled data. Consider the Omniglot meta classification task. If we have unlabeled data for all classes in the current problem, but labeled data only for a subset of the classes, a MAP-based model won't explicitly represent the "cluster-based" label assignments on which this task is based. A "Bayesian" set2func model could express this property of the underlying task distribution through the classifiers assigned high likelihood by its cross-task prior and per-task posteriors. The role of active learning, in the setting of Bayesian metalearning, is to collect labeled examples for the current problem in a way that maximally reduces entropy in the posterior distribution over classifiers. Having an explicit representation of the relevant posterior should permit more effective active learning, and adapting the prior to the task distribution should permit tighter, more-accurate per-task posteriors. Conversely, encouraging the set2func model to represent its prior and posteriors in a way that facilitates active learning may improve the quality of the learned prior and posteriors. Developing models which perform well in the few-shot setting (metalearning), and which efficiently collect information to improve task performance (active learning), will be critical to success in settings like "life-long learning". This talk discusses these themes, and presents some related concrete results.

1 1

[Tea Talk] Ioannis Mitliagkas, Fri Sep 29, 10:30AM, AA6214
by Michael Noukhovitch 29 Sep '17

29 Sep '17

This week we have a cool new MILA professor, *Ioannis Mitliagkas,* giving a talk this *Friday Sep 29* at a new, earlier time: *10:30AM* in room *AA6214*. If time permits, he'll also be giving details of his winter course on topics in AI! Hope to see you all there! Don't forget about the earlier time!! - Michael *KEYWORDS* optimization, YellowFin, large scale DL, GAN stabilization *TITLE* Understanding momentum dynamics for faster training, better scaling, and easier tuning *ABSTRACT* This talk revolves around Polyak’s momentum gradient descent method, also known as ‘momentum’. Its stochastic version, momentum stochastic gradient descent (SGD), is one of the most commonly used optimization methods in deep learning. Throughout the talk we will study a number of important properties of this versatile method, and see how this understanding can be used to engineer better deep learning systems. I will first go over the basic formulation of momentum. Then I will summarize a theoretical result on a previously unknown connection between momentum dynamics and asynchronous optimization. Understanding this connection, allows us to improve the efficiency of large-scale deep learning systems. I will go over a recent collaboration with Intel and the National Energy Research Scientific Computing Center (NERSC) on a 15 PetaFLOP system consisting of 9600 nodes. Finally, I will demonstrate how analyzing the behavior of momentum on simple objectives can lead to tuning rules for its learning rate and momentum hyperparameters. Our implementation of these rules is called YellowFin and is a simple adaptive method that can handle different objectives, as well as varying asynchronous dynamics, without hand-tuning. Yellowfin often outperforms state-of-the-art adaptive methods. At the end of the talk, I will discuss some preliminary thoughts on the training dynamics of GANs and some ideas on how momentum dynamics can, again, play a key role in stabilizing adversarial training. Finally, if time permits, I will give an outline my other research interests, as well as a rough plan for my upcoming ’Topics in AI’ class. *BIO* Ioannis Mitliagkas is an assistant professor in the Department of Computer Science and Operations Research (DIRO) at the University of Montreal. Before that, he was a Postdoctoral Scholar with the Department of Statistics and Computer Science at Stanford University. He obtained his Ph.D. from the Department of Electrical and Computer Engineering at the University of Texas at Austin. His research focuses on statistical learning and inference problems, with work in efficient large-scale and distributed algorithms, theoretical and data-dependent guarantees and tuning complex systems. His recent work includes understanding and optimizing the scanning used in Gibbs sampling for inference, as well as understanding the interaction between optimization and the dynamics of large-scale learning systems.

2 1

Anirudh Goyal, September 15, AA6214, 11AM
by Michael Noukhovitch 15 Sep '17

15 Sep '17

Hi All! Dima is going on an internship so I'll be organizing tea talks for the next couple of months! This week, our very own *Anirudh* will present his NIPS paper (congrats btw!) it will be at *AA6214 on Friday September 15th, 11AM *Hope to see everyone there! *Title:* Z-Forcing: Training Stochastic Recurrent Networks *Abstract:* Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We propose a novel sequential latent variable model unifying successful ideas from recently proposed architectures: each step in the sequence is associated with a latent variable that is used to condition the recurrent dynamics for future steps. Training is performed with amortised variational inference where the approximate posterior is augmented with a RNN that runs backward through the sequence. In addition to maximizing the variational lower bound, we ease training of the latent variables by adding an auxiliary cost which forces them to reconstruct the state of the backward recurrent network. This provides the latent variables with a task-independent objective that enhances the performance of the overall model. Although being conceptually simple, our model achieves state-of-the-art results on standard speech benchmarks such as TIMIT and BLIZZARD and competitive performance on sequential MNIST. Finally, we apply our model on language modeling in the IMDB dataset. The auxiliary cost is crucial for learning interpretable latent variables. Michael

2 2

[TeaTalk] Karan Grewal, September 8, 13:45, AA6214
by Dzmitry Bahdanau 08 Sep '17

08 Sep '17

Hi all, Our next speaker is Karan Grewal, who is currently an intern at MILA. The talk will take place at *AA6214, at 13:45, September 8*. Hope to see many of you there! *Title: *Variance Regularizing Adversarial Learning *Abstract:* Generative Adversarial Networks have become a breakthrough towards generating synthetic data in recent years, however many problems may arise during training which result in the generator being unable to learn the desired data distribution/manifold. Most notably, the discriminator often overpowers the generator, which leads to a poor training signal and vanishing gradients. We study the effects of the discriminator's unnormalized output distribution on the generator's ability to learn. To combat this problem, we propose Variance Regularizing Adversarial Learning -- regularizing the discriminator's unnormalized output distribution to fit a mixture of gaussians, yielding higher variance and hence a lipschitz-like discriminator function. To do this, we provide two methods: (1) using the KL-divergence as a penalty in the discriminator's loss and (2) playing meta-adversarial games to force the discriminator to fit a MoG. We show that our models are robust in the presence of high training ratios and compare them to other variants of GANs. *Bio:* Karan Grewal is an intern at MILA and a senior undergraduate student at the University of Toronto. He is working on improving and stabilizing generative models with Devon Hjelm and Yoshua Bengio. Previously, he worked on applying machine learning to social contexts. Dima

1 3

predoctoral thesis proposal, Taesup Kim, Tuesday 5 Sept, 15h30
by Yoshua Bengio 03 Sep '17

03 Sep '17

Hello, Taesup Kim, MILA PhD candidate, will present his predoctoral subject proposal Tuesday September 5 at 15h30 (room 3395, PAA): *Title: Dynamic Neural Networks: Learning to Adapt to DataAbstractRecent advances in deep neural networks have shown significant results on several machine learning and pattern recognition problems. While these achievements are impressive that some are already working on real-world products, there still exists a lot of potential innovations inspired by human behaviors and we cover in this proposal some related approaches to imitate human using deep neural networks involving both generative and discriminative models. Especially, we introduce some dynamic neural networks having dynamic modules that operate differently and adaptively over different data to improve both the optimization process and the performance. Related future works are also presented to show how the dynamic neural networks can be applied broadly over different domains and applications.* Please come and learn about his research and support him! *-- Yoshua*

1 0

predoctoral thesis proposal, Chinnadhurai Sankar, Tuesday 5 Sept, 13h30
by Yoshua Bengio 03 Sep '17

03 Sep '17

Hello, Chinnadhurai Sankar, MILA PhD candidate, will present his predoctoral subject proposal Tuesday September 5 at 13h30 (room 3395, PAA): *Title: Problems in Non-Goal Oriented Dialog Systems* *Abstract:* *This predoctoral presentation will mainly focus on data-driven approaches to non-goal oriented dialog systems research. I shall talk about two dialog systems projects that I have been part of. First, I shall propose a reinforcement learning based response selection model for dialogue management. The proposed method was implemented & tested in the Amazon Alexa Competition and shown to demonstrate superior performance when compared to traditional heuristics based approaches and supervised learning based approaches. **Next, I shall introduce a hybrid retrieval based generative dialog system architecture which is designed to provide a sense of specificity in the generated responses.* Please come and learn about his research and support him! *-- Yoshua*

1 0

Aller à la page :

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Lisa_seminaires